[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:39:50 dist-test-slave-ss4n krb5kdc[22212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:39:50 dist-test-slave-ss4n krb5kdc[22212](info): set up 2 sockets
Aug 10 03:39:50 dist-test-slave-ss4n krb5kdc[22212](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 03:39:51.221078 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.005s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:39:51 dist-test-slave-ss4n krb5kdc[22212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:39:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:39:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:39:51.361958 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:46687
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:36617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:46687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:39:51.479039 22225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:39:51.479396 22225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:39:51.479475 22225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:39:51.483844 22225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:39:51.483963 22225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:39:51.484000 22225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:39:51.484040 22225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:39:51.484303 22225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:39:51.490365 22225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:36617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:46687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:46687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:39:51.491616 22225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:39:51.492821 22225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:39:51.504134 22231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:51.504151 22225 server_base.cc:1034] running on GCE node
W20260810 03:39:51.504108 22230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:51.504199 22233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:51.505105 22225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:39:51.506870 22225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:39:51.508132 22225 hybrid_clock.cc:648] HybridClock initialized: now 1786333191508138 us; error 71 us; skew 500 ppm
Aug 10 03:39:51 dist-test-slave-ss4n krb5kdc[22212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333191, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:39:51.511612 22225 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:39:51.512693 22225 webserver.cc:460] Webserver started at http://127.21.175.126:43951/ using document root <none> and password file <none>
I20260810 03:39:51.513469 22225 fs_manager.cc:359] Metadata directory not provided
I20260810 03:39:51.513628 22225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:39:51.513926 22225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:39:51.516420 22225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "0252f3ac93e14039abf24a14b97ed0ec"
format_stamp: "Formatted at 2026-08-10 03:39:51 on dist-test-slave-ss4n"
server_key: "f79a9e4fb822286531524db7114f905b"
server_key_iv: "13d8f63aa5941a27e8c9a064b260e6b8"
server_key_version: "encryptionkey@0"
I20260810 03:39:51.517081 22225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "0252f3ac93e14039abf24a14b97ed0ec"
format_stamp: "Formatted at 2026-08-10 03:39:51 on dist-test-slave-ss4n"
server_key: "f79a9e4fb822286531524db7114f905b"
server_key_iv: "13d8f63aa5941a27e8c9a064b260e6b8"
server_key_version: "encryptionkey@0"
I20260810 03:39:51.521970 22225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260810 03:39:51.526178 22240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:51.528018 22225 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:39:51.528232 22225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "0252f3ac93e14039abf24a14b97ed0ec"
format_stamp: "Formatted at 2026-08-10 03:39:51 on dist-test-slave-ss4n"
server_key: "f79a9e4fb822286531524db7114f905b"
server_key_iv: "13d8f63aa5941a27e8c9a064b260e6b8"
server_key_version: "encryptionkey@0"
I20260810 03:39:51.528388 22225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:39:51.542874 22225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:39:51.546206 22225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:39:51.546487 22225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:39:51.555261 22225 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:46687
I20260810 03:39:51.555330 22296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:46687 every 8 connection(s)
I20260810 03:39:51.556495 22225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:39:51.559964 22297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:39:51.560570 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 22225
I20260810 03:39:51.560725 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:39:51.562180 22205 external_mini_cluster.cc:1428] Setting key ddb0b4659208024f1b78679d3b65ba71
I20260810 03:39:51.568019 22297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec: Bootstrap starting.
I20260810 03:39:51.571269 22297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec: Neither blocks nor log segments found. Creating new log.
I20260810 03:39:51.572316 22297 log.cc:824] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec: Log is configured to *not* fsync() on all Append() calls
I20260810 03:39:51.574725 22297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec: No bootstrap required, opened a new log
I20260810 03:39:51.579020 22297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0252f3ac93e14039abf24a14b97ed0ec" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46687 } }
I20260810 03:39:51.579308 22297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:39:51.579378 22297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0252f3ac93e14039abf24a14b97ed0ec, State: Initialized, Role: FOLLOWER
Aug 10 03:39:51 dist-test-slave-ss4n krb5kdc[22212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:39:51.580204 22297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0252f3ac93e14039abf24a14b97ed0ec" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46687 } }
I20260810 03:39:51.580523 22297 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:39:51.580636 22297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:39:51.580754 22297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:39:51.582075 22297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0252f3ac93e14039abf24a14b97ed0ec" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46687 } }
I20260810 03:39:51.582830 22297 leader_election.cc:302] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0252f3ac93e14039abf24a14b97ed0ec; no voters:
I20260810 03:39:51.583225 22297 leader_election.cc:288] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:39:51.583352 22302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:39:51.583581 22302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [term 1 LEADER]: Becoming Leader. State: Replica: 0252f3ac93e14039abf24a14b97ed0ec, State: Running, Role: LEADER
I20260810 03:39:51.584014 22302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0252f3ac93e14039abf24a14b97ed0ec" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46687 } }
I20260810 03:39:51.584898 22297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:39:51.586596 22304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0252f3ac93e14039abf24a14b97ed0ec. Latest consensus state: current_term: 1 leader_uuid: "0252f3ac93e14039abf24a14b97ed0ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0252f3ac93e14039abf24a14b97ed0ec" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46687 } } }
I20260810 03:39:51.586743 22304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [sys.catalog]: This master's current role is: LEADER
I20260810 03:39:51.588248 22303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0252f3ac93e14039abf24a14b97ed0ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0252f3ac93e14039abf24a14b97ed0ec" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46687 } } }
I20260810 03:39:51.588353 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:51.565839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.21.175.126:46687)
0810 03:39:51.566322 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:51.566337 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:51.567454 (+ 1117us) server_negotiation.cc:407] Connection header received
0810 03:39:51.570682 (+ 3228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:51.570698 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:51.571015 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:51.571352 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:51.573701 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:51.574699 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:51.575917 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:51.576396 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:51.576574 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:51.581363 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:51.581375 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:51.581392 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:51.581427 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:51.583715 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:51.584636 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:51.584643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:51.584649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:51.584820 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:51.585383 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:51.585386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:51.585388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:51.586123 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:51.586347 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:51.587049 (+ 702us) server_negotiation.cc:299] Negotiation successful
0810 03:39:51.587341 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":175,"threads_started":1}
I20260810 03:39:51.588372 22303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec [sys.catalog]: This master's current role is: LEADER
I20260810 03:39:51.588937 22308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:39:51.592672 22308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:39:51.600232 22308 catalog_manager.cc:1347] Generated new cluster ID: 5265ea9ad36e4f7b822a6b40d8d93beb
I20260810 03:39:51.600320 22308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:39:51.636906 22308 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:39:51.638327 22308 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:39:51.657744 22308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0252f3ac93e14039abf24a14b97ed0ec: Generated new TSK 0
I20260810 03:39:51.658773 22308 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:39:51.704990 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46687
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:36617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:39:51.836050 22323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:39:51.836397 22323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:39:51.836457 22323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:39:51.840776 22323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:39:51.840847 22323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:39:51.840909 22323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:39:51.846463 22323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:36617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46687
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:39:51.847599 22323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:39:51.848613 22323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:39:51.859274 22329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:51.859275 22331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:51.859275 22328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:51.859402 22323 server_base.cc:1034] running on GCE node
I20260810 03:39:51.860055 22323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:39:51.860730 22323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:39:51.861951 22323 hybrid_clock.cc:648] HybridClock initialized: now 1786333191861912 us; error 52 us; skew 500 ppm
Aug 10 03:39:51 dist-test-slave-ss4n krb5kdc[22212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333191, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:39:51.865144 22323 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:39:51.866153 22323 webserver.cc:460] Webserver started at http://127.21.175.65:33755/ using document root <none> and password file <none>
I20260810 03:39:51.866820 22323 fs_manager.cc:359] Metadata directory not provided
I20260810 03:39:51.866890 22323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:39:51.867146 22323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:39:51.869302 22323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "dc88b7413a0c476aaec5840f5d68de4e"
format_stamp: "Formatted at 2026-08-10 03:39:51 on dist-test-slave-ss4n"
server_key: "4054892b996f9e44888655655618d62a"
server_key_iv: "69c0e6feda3519747879ef091d02c90d"
server_key_version: "encryptionkey@0"
I20260810 03:39:51.869978 22323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "dc88b7413a0c476aaec5840f5d68de4e"
format_stamp: "Formatted at 2026-08-10 03:39:51 on dist-test-slave-ss4n"
server_key: "4054892b996f9e44888655655618d62a"
server_key_iv: "69c0e6feda3519747879ef091d02c90d"
server_key_version: "encryptionkey@0"
I20260810 03:39:51.874130 22323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:39:51.876868 22338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:51.878335 22323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:39:51.878463 22323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "dc88b7413a0c476aaec5840f5d68de4e"
format_stamp: "Formatted at 2026-08-10 03:39:51 on dist-test-slave-ss4n"
server_key: "4054892b996f9e44888655655618d62a"
server_key_iv: "69c0e6feda3519747879ef091d02c90d"
server_key_version: "encryptionkey@0"
I20260810 03:39:51.878590 22323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:39:51.895445 22323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:39:51.900146 22323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:39:51.900393 22323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:39:51.901222 22323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:39:51.902300 22323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:39:51.902356 22323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:51.902402 22323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:39:51.902467 22323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:51.913580 22323 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:39655
I20260810 03:39:51.913676 22456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:39655 every 8 connection(s)
I20260810 03:39:51.915192 22323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:39:51.923866 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 22323
I20260810 03:39:51.924026 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:39:51.924341 22205 external_mini_cluster.cc:1428] Setting key 6a7ea301b345b46ea2ac7f4f7c32fc00
Aug 10 03:39:51 dist-test-slave-ss4n krb5kdc[22212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333191, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:39:51.933696 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:51.918130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:58945 (local address 127.21.175.126:46687)
0810 03:39:51.918323 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:51.918329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:51.920039 (+ 1710us) server_negotiation.cc:407] Connection header received
0810 03:39:51.921392 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:51.921397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:51.921474 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:51.921624 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:51.922897 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:51.923471 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:51.925201 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:51.925595 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:51.925711 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:51.928991 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:51.928997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:51.929004 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:51.929043 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:51.931034 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:51.931618 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:51.931625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:51.931630 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:51.931744 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:51.932174 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:51.932181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:51.932185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:51.932505 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:51.932617 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:51.933324 (+ 707us) server_negotiation.cc:299] Negotiation successful
0810 03:39:51.933468 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":79}
I20260810 03:39:51.934568 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:51.918718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46687 (local address 127.21.175.65:58945)
0810 03:39:51.919807 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:51.919877 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:51.921095 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:51.921775 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:51.921782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:51.922214 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:39:51.922741 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:51.922756 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:51.923628 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:51.923635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:51.925062 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:51.925077 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:51.925753 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:51.925756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:51.926008 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:51.926778 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:39:51.926809 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:39:51.928764 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:39:51.931173 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:51.931179 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:51.931194 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:51.931481 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:51.931887 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:51.931891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:51.931894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:51.932085 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:51.932707 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:39:51.932713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:39:51.933020 (+ 307us) client_negotiation.cc:769] Sending connection context
0810 03:39:51.933266 (+ 246us) client_negotiation.cc:240] Negotiation successful
0810 03:39:51.933663 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"thread_start_us":136,"threads_started":1}
I20260810 03:39:51.936235 22458 heartbeater.cc:347] Connected to a master server at 127.21.175.126:46687
I20260810 03:39:51.936554 22458 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
I20260810 03:39:51.937279 22458 heartbeater.cc:511] Master 127.21.175.126:46687 requested a full tablet report, sending...
I20260810 03:39:51.939473 22258 ts_manager.cc:194] Registered new tserver with Master: dc88b7413a0c476aaec5840f5d68de4e (127.21.175.65:39655)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
I20260810 03:39:51.941203 22258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:58945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:39:51.961423 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46687
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:36617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:39:52.080646 22463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:39:52.080938 22463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:39:52.080986 22463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:39:52.085410 22463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:39:52.085548 22463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:39:52.085651 22463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:39:52.091836 22463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:36617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46687
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:39:52.093032 22463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:39:52.094241 22463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:39:52.105037 22471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:52.105222 22463 server_base.cc:1034] running on GCE node
W20260810 03:39:52.105108 22468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:52.105108 22469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:52.105818 22463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:39:52.106498 22463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:39:52.107699 22463 hybrid_clock.cc:648] HybridClock initialized: now 1786333192107681 us; error 28 us; skew 500 ppm
Aug 10 03:39:52 dist-test-slave-ss4n krb5kdc[22212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333192, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:39:52.110809 22463 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:39:52.111841 22463 webserver.cc:460] Webserver started at http://127.21.175.66:41895/ using document root <none> and password file <none>
I20260810 03:39:52.112469 22463 fs_manager.cc:359] Metadata directory not provided
I20260810 03:39:52.112536 22463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:39:52.112773 22463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:39:52.115091 22463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "c67417b2e831481f84bb358cef0a528f"
format_stamp: "Formatted at 2026-08-10 03:39:52 on dist-test-slave-ss4n"
server_key: "7b7b567fc4bef0540eb97dfcd92e8640"
server_key_iv: "60764dfbb5d94769bbcb33646a2a11d5"
server_key_version: "encryptionkey@0"
I20260810 03:39:52.115707 22463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "c67417b2e831481f84bb358cef0a528f"
format_stamp: "Formatted at 2026-08-10 03:39:52 on dist-test-slave-ss4n"
server_key: "7b7b567fc4bef0540eb97dfcd92e8640"
server_key_iv: "60764dfbb5d94769bbcb33646a2a11d5"
server_key_version: "encryptionkey@0"
I20260810 03:39:52.119745 22463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:39:52.122587 22478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:52.123967 22463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:39:52.124116 22463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "c67417b2e831481f84bb358cef0a528f"
format_stamp: "Formatted at 2026-08-10 03:39:52 on dist-test-slave-ss4n"
server_key: "7b7b567fc4bef0540eb97dfcd92e8640"
server_key_iv: "60764dfbb5d94769bbcb33646a2a11d5"
server_key_version: "encryptionkey@0"
I20260810 03:39:52.124243 22463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:39:52.143240 22463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:39:52.146476 22463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:39:52.146692 22463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:39:52.147320 22463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:39:52.148178 22463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:39:52.148224 22463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:52.148293 22463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:39:52.148331 22463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:52.158540 22463 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:41365
I20260810 03:39:52.158568 22596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:41365 every 8 connection(s)
I20260810 03:39:52.159507 22463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:39:52.159832 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 22463
I20260810 03:39:52.159945 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:39:52.160246 22205 external_mini_cluster.cc:1428] Setting key 51517c55ee94da7e249357d6f304ac6a
Aug 10 03:39:52 dist-test-slave-ss4n krb5kdc[22212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333192, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
I20260810 03:39:52.175657 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:52.161569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:39751 (local address 127.21.175.126:46687)
0810 03:39:52.161741 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:52.161746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:52.163403 (+ 1657us) server_negotiation.cc:407] Connection header received
0810 03:39:52.164341 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:52.164348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:52.164410 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:52.164520 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:52.165840 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:52.166753 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:52.167918 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:52.168355 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:52.168492 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:52.171083 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:52.171111 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:52.171117 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:52.171148 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:52.172808 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:52.173417 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:52.173420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:52.173421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:52.173625 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:52.174063 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:52.174068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:52.174070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:52.174399 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:52.174606 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:52.175340 (+ 734us) server_negotiation.cc:299] Negotiation successful
0810 03:39:52.175521 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:39:52.176738 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:52.162055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46687 (local address 127.21.175.66:39751)
0810 03:39:52.163219 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:52.163272 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:52.164130 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:52.164726 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:52.164732 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:52.165092 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:39:52.165686 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:52.165702 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:52.166890 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:52.166893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:52.167788 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:52.167797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:52.168507 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:52.168510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:52.168695 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:52.169341 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:39:52.169358 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:39:52.170904 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:39:52.172928 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:52.172933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:52.172947 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:52.173296 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:52.173712 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:52.173715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:52.173716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:52.173897 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:52.174664 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:39:52.174671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:39:52.175031 (+ 360us) client_negotiation.cc:769] Sending connection context
0810 03:39:52.175272 (+ 241us) client_negotiation.cc:240] Negotiation successful
0810 03:39:52.175676 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"thread_start_us":746,"threads_started":1}
I20260810 03:39:52.178210 22597 heartbeater.cc:347] Connected to a master server at 127.21.175.126:46687
I20260810 03:39:52.178479 22597 heartbeater.cc:464] Registering TS with master...
I20260810 03:39:52.179028 22597 heartbeater.cc:511] Master 127.21.175.126:46687 requested a full tablet report, sending...
I20260810 03:39:52.180120 22258 ts_manager.cc:194] Registered new tserver with Master: c67417b2e831481f84bb358cef0a528f (127.21.175.66:41365)
I20260810 03:39:52.180713 22258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:39751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:39:52.193223 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46687
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:36617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:39:52.334528 22602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:39:52.334833 22602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:39:52.334882 22602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:39:52.339089 22602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:39:52.339159 22602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:39:52.339222 22602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:39:52.344754 22602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:36617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46687
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:39:52.345883 22602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:39:52.346882 22602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:39:52.357252 22608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:52.357261 22611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:52.357261 22609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:52.357371 22602 server_base.cc:1034] running on GCE node
I20260810 03:39:52.358037 22602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:39:52.358764 22602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:39:52.359942 22602 hybrid_clock.cc:648] HybridClock initialized: now 1786333192359922 us; error 35 us; skew 500 ppm
Aug 10 03:39:52 dist-test-slave-ss4n krb5kdc[22212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333192, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:39:52.363225 22602 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:39:52.364225 22602 webserver.cc:460] Webserver started at http://127.21.175.67:34641/ using document root <none> and password file <none>
I20260810 03:39:52.364830 22602 fs_manager.cc:359] Metadata directory not provided
I20260810 03:39:52.364892 22602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:39:52.365125 22602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:39:52.367355 22602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "926e77c9236a4df188f74c97cc32808e"
format_stamp: "Formatted at 2026-08-10 03:39:52 on dist-test-slave-ss4n"
server_key: "60425fbab96e55f84f9b6505a4719c2f"
server_key_iv: "ce2813fb01f7712a122aa14c6cbafcca"
server_key_version: "encryptionkey@0"
I20260810 03:39:52.367973 22602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "926e77c9236a4df188f74c97cc32808e"
format_stamp: "Formatted at 2026-08-10 03:39:52 on dist-test-slave-ss4n"
server_key: "60425fbab96e55f84f9b6505a4719c2f"
server_key_iv: "ce2813fb01f7712a122aa14c6cbafcca"
server_key_version: "encryptionkey@0"
I20260810 03:39:52.371992 22602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 03:39:52.374540 22619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:52.375838 22602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:39:52.375983 22602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "926e77c9236a4df188f74c97cc32808e"
format_stamp: "Formatted at 2026-08-10 03:39:52 on dist-test-slave-ss4n"
server_key: "60425fbab96e55f84f9b6505a4719c2f"
server_key_iv: "ce2813fb01f7712a122aa14c6cbafcca"
server_key_version: "encryptionkey@0"
I20260810 03:39:52.376116 22602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:39:52.394466 22602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:39:52.397887 22602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:39:52.398114 22602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:39:52.398739 22602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:39:52.399611 22602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:39:52.399684 22602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:52.399757 22602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:39:52.399801 22602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:52.409647 22602 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:33141
I20260810 03:39:52.409675 22741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:33141 every 8 connection(s)
I20260810 03:39:52.410645 22602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:39:52.411640 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 22602
I20260810 03:39:52.411746 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:39:52.412065 22205 external_mini_cluster.cc:1428] Setting key 4a68759093447fd265b14f2f8e5bb605
Aug 10 03:39:52 dist-test-slave-ss4n krb5kdc[22212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333192, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:39:52.425401 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:52.412714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:43841 (local address 127.21.175.126:46687)
0810 03:39:52.412890 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:52.412895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:52.414341 (+ 1446us) server_negotiation.cc:407] Connection header received
0810 03:39:52.415503 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:52.415506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:52.415559 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:52.415632 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:52.416863 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:52.417402 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:52.418634 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:52.418895 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:52.419006 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:52.421641 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:52.421644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:52.421646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:52.421674 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:52.423366 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:52.423852 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:52.423855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:52.423857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:52.423939 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:52.424261 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:52.424263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:52.424264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:52.424455 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:52.424543 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:52.425086 (+ 543us) server_negotiation.cc:299] Negotiation successful
0810 03:39:52.425213 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:39:52.426275 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:52.413238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46687 (local address 127.21.175.67:43841)
0810 03:39:52.414192 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:52.414252 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:52.415306 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:52.415802 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:52.415809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:52.416196 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:39:52.416727 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:52.416742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:52.417569 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:52.417572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:52.418475 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:52.418484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:52.419052 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:52.419054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:52.419226 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:52.419854 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:39:52.419871 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:39:52.421401 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:39:52.423470 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:52.423475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:52.423488 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:52.423751 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:52.424018 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:52.424020 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:52.424021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:52.424170 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:52.424550 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:39:52.424554 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:39:52.424802 (+ 248us) client_negotiation.cc:769] Sending connection context
0810 03:39:52.424957 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 03:39:52.425240 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":199,"threads_started":1}
I20260810 03:39:52.427511 22742 heartbeater.cc:347] Connected to a master server at 127.21.175.126:46687
I20260810 03:39:52.427764 22742 heartbeater.cc:464] Registering TS with master...
I20260810 03:39:52.428336 22742 heartbeater.cc:511] Master 127.21.175.126:46687 requested a full tablet report, sending...
I20260810 03:39:52.429466 22258 ts_manager.cc:194] Registered new tserver with Master: 926e77c9236a4df188f74c97cc32808e (127.21.175.67:33141)
I20260810 03:39:52.430114 22258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:43841
I20260810 03:39:52.439764 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:39:52 dist-test-slave-ss4n krb5kdc[22212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333192, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:39:52 dist-test-slave-ss4n krb5kdc[22212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333192, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.175.65@KRBTEST.COM
I20260810 03:39:52.470148 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:52.459196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.21.175.65:39655)
0810 03:39:52.459601 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:52.459610 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:52.459641 (+ 31us) server_negotiation.cc:407] Connection header received
0810 03:39:52.459786 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:52.459802 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:52.460032 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:52.460208 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:52.461008 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:52.461955 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:52.462922 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:52.463352 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:52.463477 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:52.466229 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:52.466239 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:52.466255 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:52.466283 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:52.468015 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:52.468530 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:52.468536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:52.468538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:52.468630 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:52.468944 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:52.468949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:52.468951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:52.469254 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:52.469566 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:52.469773 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 03:39:52.469959 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":95,"threads_started":1}
W20260810 03:39:52.471294 22368 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34426
Aug 10 03:39:52 dist-test-slave-ss4n krb5kdc[22212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:39:52 dist-test-slave-ss4n krb5kdc[22212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.65@KRBTEST.COM
I20260810 03:39:52.501189 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:52.491936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34438 (local address 127.21.175.65:39655)
0810 03:39:52.492150 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:52.492153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:52.492240 (+ 87us) server_negotiation.cc:407] Connection header received
0810 03:39:52.492363 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:52.492365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:52.492410 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:52.492486 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:52.493323 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:52.493887 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:52.494769 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:52.494992 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:52.495079 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:52.497790 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:52.497794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:52.497796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:52.497823 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:52.499377 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:52.499838 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:52.499841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:52.499842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:52.499932 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:52.500215 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:52.500217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:52.500219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:52.500404 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:52.500520 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:52.500803 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 03:39:52.500951 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260810 03:39:52.502547 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 22323
I20260810 03:39:52.512583 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 22463
I20260810 03:39:52.521355 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 22602
I20260810 03:39:52.533047 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 22225
2026-08-10T03:39:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2496 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:39:52 dist-test-slave-ss4n krb5kdc[22758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:39:52 dist-test-slave-ss4n krb5kdc[22758](info): set up 2 sockets
Aug 10 03:39:52 dist-test-slave-ss4n krb5kdc[22758](info): commencing operation
krb5kdc: starting...
W20260810 03:39:53.633435 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:39:53 dist-test-slave-ss4n krb5kdc[22758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:39:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:39:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:39:53.774827 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:36695
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:36695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:39:53.891762 22771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:39:53.892115 22771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:39:53.892196 22771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:39:53.896351 22771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:39:53.896471 22771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:39:53.896512 22771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:39:53.896561 22771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:39:53.896595 22771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:39:53.902518 22771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:36695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:36695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:39:53.903770 22771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:39:53.904891 22771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:39:53.914255 22779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:53.914273 22776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:53.914881 22771 server_base.cc:1034] running on GCE node
W20260810 03:39:53.917528 22777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:53.918330 22771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:39:53.919973 22771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:39:53.921203 22771 hybrid_clock.cc:648] HybridClock initialized: now 1786333193921162 us; error 57 us; skew 500 ppm
Aug 10 03:39:53 dist-test-slave-ss4n krb5kdc[22758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333193, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:39:53.925206 22771 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:39:53.926265 22771 webserver.cc:460] Webserver started at http://127.21.175.126:37535/ using document root <none> and password file <none>
I20260810 03:39:53.926895 22771 fs_manager.cc:359] Metadata directory not provided
I20260810 03:39:53.927003 22771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:39:53.927255 22771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:39:53.929457 22771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "ef11e615d1d24814832a94f9d64a6a1a"
format_stamp: "Formatted at 2026-08-10 03:39:53 on dist-test-slave-ss4n"
server_key: "c97f0a77a1a1ed14caaa6124a49b8ee4"
server_key_iv: "2df22976387e99fba3b60a9418e3d3f9"
server_key_version: "encryptionkey@0"
I20260810 03:39:53.930176 22771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "ef11e615d1d24814832a94f9d64a6a1a"
format_stamp: "Formatted at 2026-08-10 03:39:53 on dist-test-slave-ss4n"
server_key: "c97f0a77a1a1ed14caaa6124a49b8ee4"
server_key_iv: "2df22976387e99fba3b60a9418e3d3f9"
server_key_version: "encryptionkey@0"
I20260810 03:39:53.934299 22771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 03:39:53.936820 22786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:53.938234 22771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:39:53.938392 22771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "ef11e615d1d24814832a94f9d64a6a1a"
format_stamp: "Formatted at 2026-08-10 03:39:53 on dist-test-slave-ss4n"
server_key: "c97f0a77a1a1ed14caaa6124a49b8ee4"
server_key_iv: "2df22976387e99fba3b60a9418e3d3f9"
server_key_version: "encryptionkey@0"
I20260810 03:39:53.938524 22771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:39:53.961335 22771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:39:53.964669 22771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:39:53.964888 22771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:39:53.973241 22771 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:36695
I20260810 03:39:53.973277 22839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:36695 every 8 connection(s)
I20260810 03:39:53.974334 22771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:39:53.977155 22840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:39:53.982939 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 22771
I20260810 03:39:53.983035 22840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a: Bootstrap starting.
I20260810 03:39:53.983103 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:39:53.983382 22205 external_mini_cluster.cc:1428] Setting key e355205d8b8bc73ee0804b0e8eb1a4ce
I20260810 03:39:53.985981 22840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a: Neither blocks nor log segments found. Creating new log.
I20260810 03:39:53.986937 22840 log.cc:824] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a: Log is configured to *not* fsync() on all Append() calls
I20260810 03:39:53.989372 22840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a: No bootstrap required, opened a new log
I20260810 03:39:53.993566 22840 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef11e615d1d24814832a94f9d64a6a1a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36695 } }
Aug 10 03:39:53 dist-test-slave-ss4n krb5kdc[22758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:39:53.993870 22840 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:39:53.993938 22840 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef11e615d1d24814832a94f9d64a6a1a, State: Initialized, Role: FOLLOWER
I20260810 03:39:53.994493 22840 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef11e615d1d24814832a94f9d64a6a1a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36695 } }
I20260810 03:39:53.994652 22840 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:39:53.994743 22840 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:39:53.994851 22840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:39:53.995795 22840 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef11e615d1d24814832a94f9d64a6a1a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36695 } }
I20260810 03:39:53.996244 22840 leader_election.cc:302] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef11e615d1d24814832a94f9d64a6a1a; no voters:
I20260810 03:39:53.996682 22840 leader_election.cc:288] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:39:53.996831 22845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:39:53.997092 22845 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [term 1 LEADER]: Becoming Leader. State: Replica: ef11e615d1d24814832a94f9d64a6a1a, State: Running, Role: LEADER
I20260810 03:39:53.997416 22845 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef11e615d1d24814832a94f9d64a6a1a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36695 } }
I20260810 03:39:53.998203 22840 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:39:53.999301 22846 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef11e615d1d24814832a94f9d64a6a1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef11e615d1d24814832a94f9d64a6a1a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36695 } } }
I20260810 03:39:53.999455 22846 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [sys.catalog]: This master's current role is: LEADER
I20260810 03:39:53.999782 22847 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef11e615d1d24814832a94f9d64a6a1a. Latest consensus state: current_term: 1 leader_uuid: "ef11e615d1d24814832a94f9d64a6a1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef11e615d1d24814832a94f9d64a6a1a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36695 } } }
I20260810 03:39:53.999893 22847 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a [sys.catalog]: This master's current role is: LEADER
I20260810 03:39:54.001091 22854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:39:54.001425 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:53.985022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34412 (local address 127.21.175.126:36695)
0810 03:39:53.985509 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:53.985530 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:53.985574 (+ 44us) server_negotiation.cc:407] Connection header received
0810 03:39:53.986524 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:53.986550 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:53.987022 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:53.987433 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:53.988315 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:53.989847 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:53.990822 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:53.991531 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:53.991713 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:53.994542 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:53.994548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:53.994562 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:53.994593 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:53.997097 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:53.997739 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:53.997746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:53.997752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:53.997905 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:53.998288 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:53.998294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:53.998296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:53.998869 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:53.999907 (+ 1038us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:54.000094 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 03:39:54.000454 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":135,"threads_started":1}
I20260810 03:39:54.005096 22854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:39:54.011184 22854 catalog_manager.cc:1347] Generated new cluster ID: f07530d6b02e4bb4ac0e6ae322d78962
I20260810 03:39:54.011272 22854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:39:54.044816 22854 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:39:54.046280 22854 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:39:54.054422 22854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef11e615d1d24814832a94f9d64a6a1a: Generated new TSK 0
I20260810 03:39:54.055361 22854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:39:54.085285 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36695
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:39:54.206899 22866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:39:54.207185 22866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:39:54.207229 22866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:39:54.211406 22866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:39:54.211472 22866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:39:54.211573 22866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:39:54.216976 22866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36695
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:39:54.218124 22866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:39:54.219197 22866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:39:54.229290 22871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:54.229314 22872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:54.229314 22874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:54.229748 22866 server_base.cc:1034] running on GCE node
I20260810 03:39:54.230162 22866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:39:54.230819 22866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:39:54.231995 22866 hybrid_clock.cc:648] HybridClock initialized: now 1786333194231973 us; error 32 us; skew 500 ppm
Aug 10 03:39:54 dist-test-slave-ss4n krb5kdc[22758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333194, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:39:54.235123 22866 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:39:54.236142 22866 webserver.cc:460] Webserver started at http://127.21.175.65:40709/ using document root <none> and password file <none>
I20260810 03:39:54.236739 22866 fs_manager.cc:359] Metadata directory not provided
I20260810 03:39:54.236833 22866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:39:54.237082 22866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:39:54.239374 22866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "6eda49dacda14ce396cd1c60215b4a82"
format_stamp: "Formatted at 2026-08-10 03:39:54 on dist-test-slave-ss4n"
server_key: "ed855902ad70cf99b6abbfee7826f8f0"
server_key_iv: "0c0c405d82f73ed0d10c72952e3c5c1b"
server_key_version: "encryptionkey@0"
I20260810 03:39:54.240015 22866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "6eda49dacda14ce396cd1c60215b4a82"
format_stamp: "Formatted at 2026-08-10 03:39:54 on dist-test-slave-ss4n"
server_key: "ed855902ad70cf99b6abbfee7826f8f0"
server_key_iv: "0c0c405d82f73ed0d10c72952e3c5c1b"
server_key_version: "encryptionkey@0"
I20260810 03:39:54.244313 22866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 03:39:54.247007 22881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:54.248265 22866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 03:39:54.248435 22866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "6eda49dacda14ce396cd1c60215b4a82"
format_stamp: "Formatted at 2026-08-10 03:39:54 on dist-test-slave-ss4n"
server_key: "ed855902ad70cf99b6abbfee7826f8f0"
server_key_iv: "0c0c405d82f73ed0d10c72952e3c5c1b"
server_key_version: "encryptionkey@0"
I20260810 03:39:54.248561 22866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:39:54.277791 22866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:39:54.280983 22866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:39:54.281214 22866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:39:54.281924 22866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:39:54.282836 22866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:39:54.282912 22866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:54.282985 22866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:39:54.283039 22866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:54.292289 22866 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:38521
I20260810 03:39:54.292310 23002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:38521 every 8 connection(s)
I20260810 03:39:54.293303 22866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:39:54.303429 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 22866
I20260810 03:39:54.303552 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:39:54.303871 22205 external_mini_cluster.cc:1428] Setting key c7af7328875ae5b39c8195c4520cd2da
Aug 10 03:39:54 dist-test-slave-ss4n krb5kdc[22758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333194, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:39:54.308521 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:54.295395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:40957 (local address 127.21.175.126:36695)
0810 03:39:54.295550 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:54.295554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:54.296486 (+ 932us) server_negotiation.cc:407] Connection header received
0810 03:39:54.298007 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:54.298011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:54.298072 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:54.298189 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:54.299598 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.300136 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.301400 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.301702 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.301815 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:54.304577 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:54.304580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:54.304583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:54.304614 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:54.306286 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:54.306868 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:54.306871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:54.306873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:54.306961 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:54.307331 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:54.307335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:54.307337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:54.307530 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:54.307658 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:54.308203 (+ 545us) server_negotiation.cc:299] Negotiation successful
0810 03:39:54.308354 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 03:39:54.309446 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:54.295838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36695 (local address 127.21.175.65:40957)
0810 03:39:54.296316 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:54.296362 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:54.297764 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:54.298430 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:54.298443 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:54.298867 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:39:54.299381 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:54.299420 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.300270 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.300281 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:54.301247 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:54.301258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.301866 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.301870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:54.302051 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:54.302717 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:39:54.302734 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:39:54.304392 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:39:54.306427 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:54.306434 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:54.306452 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:54.306749 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:54.307058 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:54.307061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:54.307063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:54.307226 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:54.307658 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:39:54.307666 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:39:54.307947 (+ 281us) client_negotiation.cc:769] Sending connection context
0810 03:39:54.308159 (+ 212us) client_negotiation.cc:240] Negotiation successful
0810 03:39:54.308529 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":120,"threads_started":1}
I20260810 03:39:54.310969 23003 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36695
I20260810 03:39:54.311257 23003 heartbeater.cc:464] Registering TS with master...
I20260810 03:39:54.311789 23003 heartbeater.cc:511] Master 127.21.175.126:36695 requested a full tablet report, sending...
I20260810 03:39:54.313421 22804 ts_manager.cc:194] Registered new tserver with Master: 6eda49dacda14ce396cd1c60215b4a82 (127.21.175.65:38521)
I20260810 03:39:54.314651 22804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:40957
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:39:54.337126 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36695
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:39:54.455943 23009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:39:54.456292 23009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:39:54.456372 23009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:39:54.460789 23009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:39:54.460891 23009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:39:54.460994 23009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:39:54.466802 23009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36695
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:39:54.468065 23009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:39:54.469201 23009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:39:54.479628 23015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:54.479609 23018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:54.479605 23016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:54.479954 23009 server_base.cc:1034] running on GCE node
I20260810 03:39:54.480401 23009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:39:54.481120 23009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:39:54.482295 23009 hybrid_clock.cc:648] HybridClock initialized: now 1786333194482277 us; error 29 us; skew 500 ppm
Aug 10 03:39:54 dist-test-slave-ss4n krb5kdc[22758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333194, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:39:54.485579 23009 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:39:54.486577 23009 webserver.cc:460] Webserver started at http://127.21.175.66:37583/ using document root <none> and password file <none>
I20260810 03:39:54.487174 23009 fs_manager.cc:359] Metadata directory not provided
I20260810 03:39:54.487272 23009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:39:54.487524 23009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:39:54.489820 23009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "53327ee2c2314f56bbb98fbe42b19b68"
format_stamp: "Formatted at 2026-08-10 03:39:54 on dist-test-slave-ss4n"
server_key: "c4e546787a273300c2f775c52b076299"
server_key_iv: "62a024cca0d869ed506a2c2c347d373e"
server_key_version: "encryptionkey@0"
I20260810 03:39:54.490458 23009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "53327ee2c2314f56bbb98fbe42b19b68"
format_stamp: "Formatted at 2026-08-10 03:39:54 on dist-test-slave-ss4n"
server_key: "c4e546787a273300c2f775c52b076299"
server_key_iv: "62a024cca0d869ed506a2c2c347d373e"
server_key_version: "encryptionkey@0"
I20260810 03:39:54.494642 23009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:39:54.497303 23026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:54.498754 23009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:39:54.498904 23009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "53327ee2c2314f56bbb98fbe42b19b68"
format_stamp: "Formatted at 2026-08-10 03:39:54 on dist-test-slave-ss4n"
server_key: "c4e546787a273300c2f775c52b076299"
server_key_iv: "62a024cca0d869ed506a2c2c347d373e"
server_key_version: "encryptionkey@0"
I20260810 03:39:54.499028 23009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:39:54.523319 23009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:39:54.527563 23009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:39:54.527756 23009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:39:54.528371 23009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:39:54.529304 23009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:39:54.529356 23009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:54.529402 23009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:39:54.529420 23009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:54.539690 23009 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:34795
I20260810 03:39:54.539711 23145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:34795 every 8 connection(s)
I20260810 03:39:54.540742 23009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:39:54.545127 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 23009
I20260810 03:39:54.545261 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:39:54.545578 22205 external_mini_cluster.cc:1428] Setting key eecf6c52500d192ae8dd5fef012d48b3
Aug 10 03:39:54 dist-test-slave-ss4n krb5kdc[22758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333194, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
I20260810 03:39:54.556556 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:54.542887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:41855 (local address 127.21.175.126:36695)
0810 03:39:54.543022 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:54.543028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:54.543970 (+ 942us) server_negotiation.cc:407] Connection header received
0810 03:39:54.544985 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:54.544989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:54.545053 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:54.545166 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:54.546692 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.547342 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.548532 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.548766 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.548903 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:54.552094 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:54.552103 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:54.552109 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:54.552146 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:54.554019 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:54.554685 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:54.554690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:54.554694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:54.554785 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:54.555186 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:54.555191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:54.555193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:54.555405 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:54.555528 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:54.556147 (+ 619us) server_negotiation.cc:299] Negotiation successful
0810 03:39:54.556295 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:39:54.557466 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:54.543308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36695 (local address 127.21.175.66:41855)
0810 03:39:54.543804 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:54.543850 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:54.544741 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:54.545362 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:54.545372 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:54.545905 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:39:54.546533 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:54.546550 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.547479 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.547483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:54.548395 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:54.548405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.549009 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.549018 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:54.549213 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:54.549945 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:39:54.549963 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:39:54.551843 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:39:54.554156 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:54.554164 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:54.554185 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:54.554543 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:54.554896 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:54.554900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:54.554902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:54.555075 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:54.555533 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:39:54.555543 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:39:54.555877 (+ 334us) client_negotiation.cc:769] Sending connection context
0810 03:39:54.556103 (+ 226us) client_negotiation.cc:240] Negotiation successful
0810 03:39:54.556506 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":128,"threads_started":1}
I20260810 03:39:54.559015 23146 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36695
I20260810 03:39:54.559357 23146 heartbeater.cc:464] Registering TS with master...
I20260810 03:39:54.559995 23146 heartbeater.cc:511] Master 127.21.175.126:36695 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
I20260810 03:39:54.561143 22804 ts_manager.cc:194] Registered new tserver with Master: 53327ee2c2314f56bbb98fbe42b19b68 (127.21.175.66:34795)
I20260810 03:39:54.561836 22804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:41855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:39:54.580334 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36695
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:39:54.706604 23151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:39:54.706919 23151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:39:54.707010 23151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:39:54.711865 23151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:39:54.711941 23151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:39:54.712042 23151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:39:54.717836 23151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36695
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:39:54.719041 23151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:39:54.720130 23151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:39:54.730633 23151 server_base.cc:1034] running on GCE node
W20260810 03:39:54.730608 23157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:54.730621 23156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:54.730650 23159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:54.731411 23151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:39:54.732129 23151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:39:54.733358 23151 hybrid_clock.cc:648] HybridClock initialized: now 1786333194733338 us; error 32 us; skew 500 ppm
Aug 10 03:39:54 dist-test-slave-ss4n krb5kdc[22758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333194, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:39:54.736464 23151 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:39:54.737579 23151 webserver.cc:460] Webserver started at http://127.21.175.67:43269/ using document root <none> and password file <none>
I20260810 03:39:54.738168 23151 fs_manager.cc:359] Metadata directory not provided
I20260810 03:39:54.738229 23151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:39:54.738406 23151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:39:54.740638 23151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "ec5673d30fa940f38a0b24a3ed063253"
format_stamp: "Formatted at 2026-08-10 03:39:54 on dist-test-slave-ss4n"
server_key: "15999da8d73b84718d39b4035cda9734"
server_key_iv: "324ab0ef3f45768b718581597a9982aa"
server_key_version: "encryptionkey@0"
I20260810 03:39:54.741214 23151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "ec5673d30fa940f38a0b24a3ed063253"
format_stamp: "Formatted at 2026-08-10 03:39:54 on dist-test-slave-ss4n"
server_key: "15999da8d73b84718d39b4035cda9734"
server_key_iv: "324ab0ef3f45768b718581597a9982aa"
server_key_version: "encryptionkey@0"
I20260810 03:39:54.745810 23151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260810 03:39:54.748605 23166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:54.750022 23151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:39:54.750161 23151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "ec5673d30fa940f38a0b24a3ed063253"
format_stamp: "Formatted at 2026-08-10 03:39:54 on dist-test-slave-ss4n"
server_key: "15999da8d73b84718d39b4035cda9734"
server_key_iv: "324ab0ef3f45768b718581597a9982aa"
server_key_version: "encryptionkey@0"
I20260810 03:39:54.750268 23151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:39:54.780053 23151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:39:54.783344 23151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:39:54.783511 23151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:39:54.784063 23151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:39:54.784885 23151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:39:54.784933 23151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:54.784976 23151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:39:54.784991 23151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:54.794034 23151 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:36521
I20260810 03:39:54.794067 23284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:36521 every 8 connection(s)
I20260810 03:39:54.795058 23151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:39:54.798452 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 23151
I20260810 03:39:54.798605 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:39:54.798966 22205 external_mini_cluster.cc:1428] Setting key 3fb3b782fd11ae5ba7139e2976f0bd1e
Aug 10 03:39:54 dist-test-slave-ss4n krb5kdc[22758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333194, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:39:54.809722 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:54.797061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:44377 (local address 127.21.175.126:36695)
0810 03:39:54.797226 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:54.797230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:54.798280 (+ 1050us) server_negotiation.cc:407] Connection header received
0810 03:39:54.799496 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:54.799499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:54.799559 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:54.799643 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:54.801411 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.802035 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.803407 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.803693 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.803787 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:54.806507 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:54.806510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:54.806512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:54.806539 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:54.807823 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:54.808304 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:54.808307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:54.808308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:54.808384 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:54.808690 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:54.808692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:54.808693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:54.808865 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:54.808940 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:54.809426 (+ 486us) server_negotiation.cc:299] Negotiation successful
0810 03:39:54.809595 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:39:54.810886 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:54.797435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36695 (local address 127.21.175.67:44377)
0810 03:39:54.798067 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:54.798141 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:54.799273 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:54.799946 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:54.799963 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:54.800576 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:39:54.801244 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:54.801265 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.802177 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.802180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:54.803282 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:54.803293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.803859 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.803863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:54.804029 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:54.804618 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:39:54.804635 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:39:54.806339 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:39:54.807934 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:54.807939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:54.807952 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:54.808216 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:54.808462 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:54.808465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:54.808466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:54.808610 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:54.808955 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:39:54.808961 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:39:54.809206 (+ 245us) client_negotiation.cc:769] Sending connection context
0810 03:39:54.809378 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 03:39:54.810028 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":161,"threads_started":1}
I20260810 03:39:54.812299 23285 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36695
I20260810 03:39:54.812625 23285 heartbeater.cc:464] Registering TS with master...
I20260810 03:39:54.813268 23285 heartbeater.cc:511] Master 127.21.175.126:36695 requested a full tablet report, sending...
I20260810 03:39:54.814369 22804 ts_manager.cc:194] Registered new tserver with Master: ec5673d30fa940f38a0b24a3ed063253 (127.21.175.67:36521)
I20260810 03:39:54.814960 22804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:44377
I20260810 03:39:54.825569 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:39:54.838420 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:54.829853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34422 (local address 127.21.175.126:36695)
0810 03:39:54.830103 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:54.830109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:54.830286 (+ 177us) server_negotiation.cc:407] Connection header received
0810 03:39:54.830573 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:54.830578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:54.830643 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:54.830752 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:54.831659 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.832273 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.833262 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.833609 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.833748 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:54.835258 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:54.835262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:54.835265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:54.835301 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:54.836594 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:54.837026 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:54.837030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:54.837032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:54.837122 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:54.837414 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:54.837417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:54.837419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:54.837730 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:54.837877 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:54.838131 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 03:39:54.838259 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260810 03:39:54.844106 22804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:39:54.847534 22804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:39:54.864630 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:54.859536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.21.175.66:34795)
0810 03:39:54.859853 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:54.859864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:54.859885 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:39:54.859985 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:54.859990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:54.860203 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:54.860412 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:39:54.860682 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.862103 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.863402 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.863942 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.864072 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:54.864247 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:54.864367 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 03:39:54.864466 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":134,"threads_started":1}
I20260810 03:39:54.864624 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:54.857838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:34795 (local address 127.0.0.1:54798)
0810 03:39:54.859163 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:54.859176 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:54.859278 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:54.860405 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:54.860409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:54.860430 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:39:54.860549 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:54.860564 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.862270 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.862274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:54.863278 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:54.863286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.864074 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.864077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:54.864185 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:54.864222 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 03:39:54.864339 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 03:39:54.864466 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1234,"thread_start_us":97,"threads_started":1}
I20260810 03:39:54.865052 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:54.857873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.21.175.67:36521)
0810 03:39:54.858337 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:54.858351 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:54.858939 (+ 588us) server_negotiation.cc:407] Connection header received
0810 03:39:54.859280 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:54.859287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:54.859502 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:54.859680 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:39:54.860025 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.861239 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.863065 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.864007 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.864116 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:54.864268 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:54.864716 (+ 448us) server_negotiation.cc:299] Negotiation successful
0810 03:39:54.864877 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":250,"threads_started":1}
I20260810 03:39:54.865110 23296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:54.857769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:36521 (local address 127.0.0.1:33092)
0810 03:39:54.858755 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:54.858817 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:54.859043 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:54.859672 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:54.859679 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:54.859705 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:39:54.859878 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:54.859886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.861454 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.861457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:54.862911 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:54.862922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.864336 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.864340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:54.864449 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:54.864477 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 03:39:54.864554 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 03:39:54.864631 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":110,"threads_started":1}
I20260810 03:39:54.866613 23297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:54.858673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:38521 (local address 127.0.0.1:45900)
0810 03:39:54.860169 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:54.860181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:54.860310 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:54.860846 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:54.860849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:54.860869 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:39:54.860982 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:54.860990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.862639 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.862646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:54.863877 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:54.863890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.866183 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.866187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:54.866283 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:54.866307 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:39:54.866391 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 03:39:54.866461 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1441,"mutex_wait_us":3,"thread_start_us":121,"threads_started":1}
I20260810 03:39:54.866818 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:54.859151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45900 (local address 127.21.175.65:38521)
0810 03:39:54.860044 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:54.860051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:54.860284 (+ 233us) server_negotiation.cc:407] Connection header received
0810 03:39:54.860500 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:54.860505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:54.860720 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:54.860872 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:39:54.861109 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.862511 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.864049 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:54.865812 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:54.865935 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:54.866112 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:54.866464 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 03:39:54.866574 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":128,"threads_started":1}
I20260810 03:39:54.868376 23075 tablet_service.cc:1467] Processing CreateTablet for tablet 65da87dc5c254398b227dcebf7b98aa9 (DEFAULT_TABLE table=test-table [id=903e04cd373a44da900d3ec5c5c753c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:39:54.868530 23214 tablet_service.cc:1467] Processing CreateTablet for tablet 65da87dc5c254398b227dcebf7b98aa9 (DEFAULT_TABLE table=test-table [id=903e04cd373a44da900d3ec5c5c753c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:39:54.869450 23075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65da87dc5c254398b227dcebf7b98aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:39:54.869704 23214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65da87dc5c254398b227dcebf7b98aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:39:54.869875 22933 tablet_service.cc:1467] Processing CreateTablet for tablet 65da87dc5c254398b227dcebf7b98aa9 (DEFAULT_TABLE table=test-table [id=903e04cd373a44da900d3ec5c5c753c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:39:54.870930 22933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65da87dc5c254398b227dcebf7b98aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:39:54.876149 23303 tablet_bootstrap.cc:492] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68: Bootstrap starting.
I20260810 03:39:54.878360 23305 tablet_bootstrap.cc:492] T 65da87dc5c254398b227dcebf7b98aa9 P 6eda49dacda14ce396cd1c60215b4a82: Bootstrap starting.
I20260810 03:39:54.878906 23303 tablet_bootstrap.cc:654] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68: Neither blocks nor log segments found. Creating new log.
I20260810 03:39:54.879305 23304 tablet_bootstrap.cc:492] T 65da87dc5c254398b227dcebf7b98aa9 P ec5673d30fa940f38a0b24a3ed063253: Bootstrap starting.
I20260810 03:39:54.880404 23303 log.cc:824] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68: Log is configured to *not* fsync() on all Append() calls
I20260810 03:39:54.880915 23305 tablet_bootstrap.cc:654] T 65da87dc5c254398b227dcebf7b98aa9 P 6eda49dacda14ce396cd1c60215b4a82: Neither blocks nor log segments found. Creating new log.
I20260810 03:39:54.881433 23304 tablet_bootstrap.cc:654] T 65da87dc5c254398b227dcebf7b98aa9 P ec5673d30fa940f38a0b24a3ed063253: Neither blocks nor log segments found. Creating new log.
I20260810 03:39:54.881798 23305 log.cc:824] T 65da87dc5c254398b227dcebf7b98aa9 P 6eda49dacda14ce396cd1c60215b4a82: Log is configured to *not* fsync() on all Append() calls
I20260810 03:39:54.882774 23303 tablet_bootstrap.cc:492] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68: No bootstrap required, opened a new log
I20260810 03:39:54.882757 23304 log.cc:824] T 65da87dc5c254398b227dcebf7b98aa9 P ec5673d30fa940f38a0b24a3ed063253: Log is configured to *not* fsync() on all Append() calls
I20260810 03:39:54.882980 23303 ts_tablet_manager.cc:1397] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 03:39:54.883875 23305 tablet_bootstrap.cc:492] T 65da87dc5c254398b227dcebf7b98aa9 P 6eda49dacda14ce396cd1c60215b4a82: No bootstrap required, opened a new log
I20260810 03:39:54.884063 23305 ts_tablet_manager.cc:1397] T 65da87dc5c254398b227dcebf7b98aa9 P 6eda49dacda14ce396cd1c60215b4a82: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:39:54.884805 23304 tablet_bootstrap.cc:492] T 65da87dc5c254398b227dcebf7b98aa9 P ec5673d30fa940f38a0b24a3ed063253: No bootstrap required, opened a new log
I20260810 03:39:54.885304 23304 ts_tablet_manager.cc:1397] T 65da87dc5c254398b227dcebf7b98aa9 P ec5673d30fa940f38a0b24a3ed063253: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260810 03:39:54.887298 23303 raft_consensus.cc:348] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5673d30fa940f38a0b24a3ed063253" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36521 } } peers { permanent_uuid: "53327ee2c2314f56bbb98fbe42b19b68" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34795 } } peers { permanent_uuid: "6eda49dacda14ce396cd1c60215b4a82" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38521 } }
I20260810 03:39:54.887657 23303 raft_consensus.cc:374] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:39:54.887725 23303 raft_consensus.cc:729] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53327ee2c2314f56bbb98fbe42b19b68, State: Initialized, Role: FOLLOWER
I20260810 03:39:54.888222 23303 consensus_queue.cc:260] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5673d30fa940f38a0b24a3ed063253" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36521 } } peers { permanent_uuid: "53327ee2c2314f56bbb98fbe42b19b68" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34795 } } peers { permanent_uuid: "6eda49dacda14ce396cd1c60215b4a82" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38521 } }
I20260810 03:39:54.888803 23305 raft_consensus.cc:348] T 65da87dc5c254398b227dcebf7b98aa9 P 6eda49dacda14ce396cd1c60215b4a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5673d30fa940f38a0b24a3ed063253" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36521 } } peers { permanent_uuid: "53327ee2c2314f56bbb98fbe42b19b68" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34795 } } peers { permanent_uuid: "6eda49dacda14ce396cd1c60215b4a82" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38521 } }
I20260810 03:39:54.888993 23146 heartbeater.cc:503] Master 127.21.175.126:36695 was elected leader, sending a full tablet report...
I20260810 03:39:54.889055 23305 raft_consensus.cc:374] T 65da87dc5c254398b227dcebf7b98aa9 P 6eda49dacda14ce396cd1c60215b4a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:39:54.889117 23305 raft_consensus.cc:729] T 65da87dc5c254398b227dcebf7b98aa9 P 6eda49dacda14ce396cd1c60215b4a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eda49dacda14ce396cd1c60215b4a82, State: Initialized, Role: FOLLOWER
I20260810 03:39:54.888926 23304 raft_consensus.cc:348] T 65da87dc5c254398b227dcebf7b98aa9 P ec5673d30fa940f38a0b24a3ed063253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5673d30fa940f38a0b24a3ed063253" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36521 } } peers { permanent_uuid: "53327ee2c2314f56bbb98fbe42b19b68" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34795 } } peers { permanent_uuid: "6eda49dacda14ce396cd1c60215b4a82" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38521 } }
I20260810 03:39:54.889226 23304 raft_consensus.cc:374] T 65da87dc5c254398b227dcebf7b98aa9 P ec5673d30fa940f38a0b24a3ed063253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:39:54.889279 23304 raft_consensus.cc:729] T 65da87dc5c254398b227dcebf7b98aa9 P ec5673d30fa940f38a0b24a3ed063253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5673d30fa940f38a0b24a3ed063253, State: Initialized, Role: FOLLOWER
I20260810 03:39:54.889349 23303 ts_tablet_manager.cc:1428] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 03:39:54.889605 23305 consensus_queue.cc:260] T 65da87dc5c254398b227dcebf7b98aa9 P 6eda49dacda14ce396cd1c60215b4a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5673d30fa940f38a0b24a3ed063253" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36521 } } peers { permanent_uuid: "53327ee2c2314f56bbb98fbe42b19b68" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34795 } } peers { permanent_uuid: "6eda49dacda14ce396cd1c60215b4a82" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38521 } }
I20260810 03:39:54.889775 23304 consensus_queue.cc:260] T 65da87dc5c254398b227dcebf7b98aa9 P ec5673d30fa940f38a0b24a3ed063253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5673d30fa940f38a0b24a3ed063253" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36521 } } peers { permanent_uuid: "53327ee2c2314f56bbb98fbe42b19b68" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34795 } } peers { permanent_uuid: "6eda49dacda14ce396cd1c60215b4a82" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38521 } }
I20260810 03:39:54.890324 23003 heartbeater.cc:503] Master 127.21.175.126:36695 was elected leader, sending a full tablet report...
I20260810 03:39:54.890679 23285 heartbeater.cc:503] Master 127.21.175.126:36695 was elected leader, sending a full tablet report...
I20260810 03:39:54.891680 23305 ts_tablet_manager.cc:1428] T 65da87dc5c254398b227dcebf7b98aa9 P 6eda49dacda14ce396cd1c60215b4a82: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260810 03:39:54.893764 23304 ts_tablet_manager.cc:1428] T 65da87dc5c254398b227dcebf7b98aa9 P ec5673d30fa940f38a0b24a3ed063253: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
W20260810 03:39:55.042089 23147 tablet.cc:2369] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:39:55.044834 23004 tablet.cc:2369] T 65da87dc5c254398b227dcebf7b98aa9 P 6eda49dacda14ce396cd1c60215b4a82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:39:55.046103 23286 tablet.cc:2369] T 65da87dc5c254398b227dcebf7b98aa9 P ec5673d30fa940f38a0b24a3ed063253: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:39:55.051641 23309 raft_consensus.cc:482] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:39:55.051885 23309 raft_consensus.cc:504] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5673d30fa940f38a0b24a3ed063253" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36521 } } peers { permanent_uuid: "53327ee2c2314f56bbb98fbe42b19b68" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34795 } } peers { permanent_uuid: "6eda49dacda14ce396cd1c60215b4a82" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38521 } }
I20260810 03:39:55.053340 23309 leader_election.cc:288] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec5673d30fa940f38a0b24a3ed063253 (127.21.175.67:36521), 6eda49dacda14ce396cd1c60215b4a82 (127.21.175.65:38521)
I20260810 03:39:55.057586 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:55.053878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:47279 (local address 127.21.175.65:38521)
0810 03:39:55.054058 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:55.054063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:55.054079 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:39:55.054140 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:55.054145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:55.054199 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:55.054294 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:39:55.054618 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:55.055334 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:55.056398 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:55.056940 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:55.057107 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:55.057164 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:55.057356 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 03:39:55.057428 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:39:55.057586 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:55.053621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:38521 (local address 127.21.175.66:47279)
0810 03:39:55.053824 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:55.053840 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:55.053955 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:55.054320 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:55.054323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:55.054349 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:39:55.054492 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:55.054502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:55.055455 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:55.055460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:55.056248 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:55.056257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:55.057104 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:55.057108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:55.057223 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:55.057243 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 03:39:55.057337 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 03:39:55.057417 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":1}
I20260810 03:39:55.057767 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:55.054006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:36521 (local address 127.21.175.66:51929)
0810 03:39:55.054321 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:55.054334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:55.054439 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:55.054703 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:55.054705 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:55.054721 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:39:55.054850 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:55.054857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:55.055653 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:55.055659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:55.056538 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:55.056549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:55.057266 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:55.057270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:55.057376 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:55.057406 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 03:39:55.057546 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 03:39:55.057630 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":116,"threads_started":1}
I20260810 03:39:55.057755 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:55.053718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:51929 (local address 127.21.175.67:36521)
0810 03:39:55.053869 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:55.053874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:55.054442 (+ 568us) server_negotiation.cc:407] Connection header received
0810 03:39:55.054542 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:55.054545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:55.054603 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:55.054686 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:39:55.054978 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:55.055526 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:55.056691 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:55.057123 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:55.057285 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:55.057339 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:55.057567 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 03:39:55.057639 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 03:39:55.058616 22955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65da87dc5c254398b227dcebf7b98aa9" candidate_uuid: "53327ee2c2314f56bbb98fbe42b19b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eda49dacda14ce396cd1c60215b4a82" is_pre_election: true
I20260810 03:39:55.058652 23238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65da87dc5c254398b227dcebf7b98aa9" candidate_uuid: "53327ee2c2314f56bbb98fbe42b19b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5673d30fa940f38a0b24a3ed063253" is_pre_election: true
I20260810 03:39:55.059070 22955 raft_consensus.cc:2454] T 65da87dc5c254398b227dcebf7b98aa9 P 6eda49dacda14ce396cd1c60215b4a82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53327ee2c2314f56bbb98fbe42b19b68 in term 0.
I20260810 03:39:55.059621 23029 leader_election.cc:302] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53327ee2c2314f56bbb98fbe42b19b68, 6eda49dacda14ce396cd1c60215b4a82; no voters:
I20260810 03:39:55.059942 23309 raft_consensus.cc:2781] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:39:55.060043 23309 raft_consensus.cc:482] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:39:55.060078 23309 raft_consensus.cc:3037] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:39:55.060819 23238 raft_consensus.cc:2454] T 65da87dc5c254398b227dcebf7b98aa9 P ec5673d30fa940f38a0b24a3ed063253 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53327ee2c2314f56bbb98fbe42b19b68 in term 0.
I20260810 03:39:55.061401 23309 raft_consensus.cc:504] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5673d30fa940f38a0b24a3ed063253" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36521 } } peers { permanent_uuid: "53327ee2c2314f56bbb98fbe42b19b68" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34795 } } peers { permanent_uuid: "6eda49dacda14ce396cd1c60215b4a82" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38521 } }
I20260810 03:39:55.061883 23309 leader_election.cc:288] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 [CANDIDATE]: Term 1 election: Requested vote from peers ec5673d30fa940f38a0b24a3ed063253 (127.21.175.67:36521), 6eda49dacda14ce396cd1c60215b4a82 (127.21.175.65:38521)
I20260810 03:39:55.062420 22955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65da87dc5c254398b227dcebf7b98aa9" candidate_uuid: "53327ee2c2314f56bbb98fbe42b19b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eda49dacda14ce396cd1c60215b4a82"
I20260810 03:39:55.062521 23238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65da87dc5c254398b227dcebf7b98aa9" candidate_uuid: "53327ee2c2314f56bbb98fbe42b19b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5673d30fa940f38a0b24a3ed063253"
I20260810 03:39:55.062654 22955 raft_consensus.cc:3037] T 65da87dc5c254398b227dcebf7b98aa9 P 6eda49dacda14ce396cd1c60215b4a82 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:39:55.062662 23238 raft_consensus.cc:3037] T 65da87dc5c254398b227dcebf7b98aa9 P ec5673d30fa940f38a0b24a3ed063253 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:39:55.063805 22955 raft_consensus.cc:2454] T 65da87dc5c254398b227dcebf7b98aa9 P 6eda49dacda14ce396cd1c60215b4a82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53327ee2c2314f56bbb98fbe42b19b68 in term 1.
I20260810 03:39:55.063819 23238 raft_consensus.cc:2454] T 65da87dc5c254398b227dcebf7b98aa9 P ec5673d30fa940f38a0b24a3ed063253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53327ee2c2314f56bbb98fbe42b19b68 in term 1.
I20260810 03:39:55.064178 23029 leader_election.cc:302] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53327ee2c2314f56bbb98fbe42b19b68, 6eda49dacda14ce396cd1c60215b4a82; no voters:
I20260810 03:39:55.064412 23309 raft_consensus.cc:2781] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:39:55.064666 23309 raft_consensus.cc:686] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 [term 1 LEADER]: Becoming Leader. State: Replica: 53327ee2c2314f56bbb98fbe42b19b68, State: Running, Role: LEADER
I20260810 03:39:55.065083 23309 consensus_queue.cc:237] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5673d30fa940f38a0b24a3ed063253" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36521 } } peers { permanent_uuid: "53327ee2c2314f56bbb98fbe42b19b68" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34795 } } peers { permanent_uuid: "6eda49dacda14ce396cd1c60215b4a82" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38521 } }
I20260810 03:39:55.068684 22803 catalog_manager.cc:5495] T 65da87dc5c254398b227dcebf7b98aa9 P 53327ee2c2314f56bbb98fbe42b19b68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53327ee2c2314f56bbb98fbe42b19b68 (127.21.175.66). New cstate: current_term: 1 leader_uuid: "53327ee2c2314f56bbb98fbe42b19b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5673d30fa940f38a0b24a3ed063253" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53327ee2c2314f56bbb98fbe42b19b68" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6eda49dacda14ce396cd1c60215b4a82" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38521 } health_report { overall_health: UNKNOWN } } }
Aug 10 03:39:55 dist-test-slave-ss4n krb5kdc[22758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.65@KRBTEST.COM
I20260810 03:39:55.112224 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:55.102179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45914 (local address 127.21.175.65:38521)
0810 03:39:55.102317 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:55.102321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:55.102508 (+ 187us) server_negotiation.cc:407] Connection header received
0810 03:39:55.102620 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:55.102623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:55.102668 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:55.102744 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:55.103690 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:55.104268 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:55.105203 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:55.105422 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:55.105613 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:55.108225 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:55.108239 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:55.108257 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:55.108286 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:55.110367 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:55.110831 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:55.110836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:55.110838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:55.110928 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:55.111166 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:55.111168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:55.111170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:55.111426 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:55.111603 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:55.111901 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 03:39:55.112063 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 03:39:55 dist-test-slave-ss4n krb5kdc[22758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333195, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:39:55 dist-test-slave-ss4n krb5kdc[22758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333195, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.175.65@KRBTEST.COM
I20260810 03:39:55.147137 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:55.132919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.21.175.65:38521)
0810 03:39:55.133129 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:55.133135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:55.133240 (+ 105us) server_negotiation.cc:407] Connection header received
0810 03:39:55.133362 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:55.133364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:55.133413 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:55.133571 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:55.134209 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:55.134758 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:55.135615 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:55.135845 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:55.135929 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:55.144006 (+ 8077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:55.144010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:55.144012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:55.144051 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:55.145370 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:55.145849 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:55.145852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:55.145853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:55.145936 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:55.146285 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:55.146288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:55.146289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:55.146490 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:55.146599 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:55.146897 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 03:39:55.147010 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Aug 10 03:39:55 dist-test-slave-ss4n krb5kdc[22758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:39:55 dist-test-slave-ss4n krb5kdc[22758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.65@KRBTEST.COM
I20260810 03:39:55.176980 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:55.167820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45934 (local address 127.21.175.65:38521)
0810 03:39:55.168015 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:55.168020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:55.168204 (+ 184us) server_negotiation.cc:407] Connection header received
0810 03:39:55.168391 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:55.168394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:55.168497 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:55.168599 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:55.169338 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:55.169942 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:55.170801 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:55.171035 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:55.171146 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:55.173724 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:55.173727 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:55.173729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:55.173755 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:55.175319 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:55.175771 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:55.175774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:55.175775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:55.175859 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:55.176117 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:55.176120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:55.176121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:55.176299 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:55.176399 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:55.176706 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 03:39:55.176836 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260810 03:39:55.178551 22912 tablet_service.cc:2908] Rejecting scan request for tablet 65da87dc5c254398b227dcebf7b98aa9: Uninitialized: clean time has not yet been initialized
I20260810 03:39:55.180977 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 22866
I20260810 03:39:55.189301 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 23009
I20260810 03:39:55.196058 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 23151
I20260810 03:39:55.205747 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 22771
2026-08-10T03:39:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2674 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:39:55 dist-test-slave-ss4n krb5kdc[23327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:39:55 dist-test-slave-ss4n krb5kdc[23327](info): set up 2 sockets
Aug 10 03:39:55 dist-test-slave-ss4n krb5kdc[23327](info): commencing operation
krb5kdc: starting...
W20260810 03:39:56.313392 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:39:56 dist-test-slave-ss4n krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:39:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:39:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:39:56.449769 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:38579
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:38579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:39:56.568849 23340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:39:56.569180 23340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:39:56.569263 23340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:39:56.573411 23340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:39:56.573540 23340 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 03:39:56.573585 23340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:39:56.573625 23340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:39:56.573665 23340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:39:56.573701 23340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:39:56.579206 23340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:38579
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:38579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:39:56.580349 23340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:39:56.581422 23340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:39:56.590345 23345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:56.590401 23348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:56.590348 23346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:56.590749 23340 server_base.cc:1034] running on GCE node
I20260810 03:39:56.591388 23340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:39:56.592559 23340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:39:56.593724 23340 hybrid_clock.cc:648] HybridClock initialized: now 1786333196593711 us; error 29 us; skew 500 ppm
Aug 10 03:39:56 dist-test-slave-ss4n krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333196, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:39:56.596514 23340 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:39:56.597427 23340 webserver.cc:460] Webserver started at http://127.21.175.126:45883/ using document root <none> and password file <none>
I20260810 03:39:56.598016 23340 fs_manager.cc:359] Metadata directory not provided
I20260810 03:39:56.598081 23340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:39:56.598251 23340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:39:56.600313 23340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "ee922360100a485eacd6de8a07db609b"
format_stamp: "Formatted at 2026-08-10 03:39:56 on dist-test-slave-ss4n"
server_key: "77973e05cc7a44c70646908d859fad93"
server_key_iv: "dd9af4dded7549e10f277bd6660aa3d5"
server_key_version: "encryptionkey@0"
I20260810 03:39:56.600845 23340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "ee922360100a485eacd6de8a07db609b"
format_stamp: "Formatted at 2026-08-10 03:39:56 on dist-test-slave-ss4n"
server_key: "77973e05cc7a44c70646908d859fad93"
server_key_iv: "dd9af4dded7549e10f277bd6660aa3d5"
server_key_version: "encryptionkey@0"
I20260810 03:39:56.605050 23340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:39:56.607956 23355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:56.609352 23340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:39:56.609465 23340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "ee922360100a485eacd6de8a07db609b"
format_stamp: "Formatted at 2026-08-10 03:39:56 on dist-test-slave-ss4n"
server_key: "77973e05cc7a44c70646908d859fad93"
server_key_iv: "dd9af4dded7549e10f277bd6660aa3d5"
server_key_version: "encryptionkey@0"
I20260810 03:39:56.609656 23340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:39:56.636271 23340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:39:56.639729 23340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:39:56.639902 23340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:39:56.649061 23340 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:38579
I20260810 03:39:56.649070 23421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:38579 every 8 connection(s)
I20260810 03:39:56.650192 23340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:39:56.653147 23422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:39:56.658461 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 23340
I20260810 03:39:56.658596 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:39:56.658859 22205 external_mini_cluster.cc:1428] Setting key 5dbd142fe6506eed2c6cbaa7afb587b9
I20260810 03:39:56.659217 23422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b: Bootstrap starting.
I20260810 03:39:56.662086 23422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b: Neither blocks nor log segments found. Creating new log.
I20260810 03:39:56.663056 23422 log.cc:824] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:39:56.665146 23422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b: No bootstrap required, opened a new log
Aug 10 03:39:56 dist-test-slave-ss4n krb5kdc[23327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:39:56.669701 23422 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee922360100a485eacd6de8a07db609b" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38579 } }
I20260810 03:39:56.670032 23422 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:39:56.670120 23422 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee922360100a485eacd6de8a07db609b, State: Initialized, Role: FOLLOWER
I20260810 03:39:56.670712 23422 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee922360100a485eacd6de8a07db609b" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38579 } }
I20260810 03:39:56.670912 23422 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:39:56.671031 23422 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:39:56.671181 23422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:39:56.672545 23422 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee922360100a485eacd6de8a07db609b" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38579 } }
I20260810 03:39:56.672961 23422 leader_election.cc:302] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee922360100a485eacd6de8a07db609b; no voters:
I20260810 03:39:56.673377 23422 leader_election.cc:288] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:39:56.673550 23427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:39:56.673882 23427 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [term 1 LEADER]: Becoming Leader. State: Replica: ee922360100a485eacd6de8a07db609b, State: Running, Role: LEADER
I20260810 03:39:56.674288 23427 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee922360100a485eacd6de8a07db609b" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38579 } }
I20260810 03:39:56.674832 23422 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:39:56.675645 23429 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee922360100a485eacd6de8a07db609b. Latest consensus state: current_term: 1 leader_uuid: "ee922360100a485eacd6de8a07db609b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee922360100a485eacd6de8a07db609b" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38579 } } }
I20260810 03:39:56.676410 23428 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee922360100a485eacd6de8a07db609b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee922360100a485eacd6de8a07db609b" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38579 } } }
I20260810 03:39:56.676502 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:56.660416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42846 (local address 127.21.175.126:38579)
0810 03:39:56.660884 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:56.660904 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:56.660948 (+ 44us) server_negotiation.cc:407] Connection header received
0810 03:39:56.661865 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:56.661894 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:56.662416 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:56.662830 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:56.663890 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:56.665114 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:56.666290 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:56.666944 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:56.667178 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:56.670379 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:56.670387 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:56.670408 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:56.670446 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:56.672791 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:56.673359 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:56.673365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:56.673372 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:56.673556 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:56.673938 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:56.673945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:56.673949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:56.674536 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:56.674705 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:56.675168 (+ 463us) server_negotiation.cc:299] Negotiation successful
0810 03:39:56.675496 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":149,"threads_started":1}
I20260810 03:39:56.676527 23428 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [sys.catalog]: This master's current role is: LEADER
I20260810 03:39:56.675884 23429 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b [sys.catalog]: This master's current role is: LEADER
I20260810 03:39:56.677017 23439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:39:56.680456 23439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:39:56.687312 23439 catalog_manager.cc:1347] Generated new cluster ID: f1edac92af8e40adb3f6a9554ba92ba7
I20260810 03:39:56.687398 23439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:39:56.696837 23439 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:39:56.697970 23439 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:39:56.703207 23439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b: Generated new TSK 0
I20260810 03:39:56.703929 23439 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:39:56.740916 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:38579
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:39:56.871320 23448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:39:56.871627 23448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:39:56.871675 23448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:39:56.875954 23448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:39:56.876021 23448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:39:56.876065 23448 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:39:56.876098 23448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:39:56.881587 23448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.175.126:38579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:39:56.882685 23448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:39:56.883719 23448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:39:56.893863 23456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:56.893880 23453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:56.893936 23448 server_base.cc:1034] running on GCE node
W20260810 03:39:56.893858 23454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:56.894536 23448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:39:56.895273 23448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:39:56.896451 23448 hybrid_clock.cc:648] HybridClock initialized: now 1786333196896431 us; error 31 us; skew 500 ppm
Aug 10 03:39:56 dist-test-slave-ss4n krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333196, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:39:56.899616 23448 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:39:56.900568 23448 webserver.cc:460] Webserver started at http://127.21.175.65:43417/ using document root <none> and password file <none>
I20260810 03:39:56.901217 23448 fs_manager.cc:359] Metadata directory not provided
I20260810 03:39:56.901288 23448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:39:56.901602 23448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:39:56.903824 23448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "e42b8e37147c4a63bc393030b20d2e97"
format_stamp: "Formatted at 2026-08-10 03:39:56 on dist-test-slave-ss4n"
server_key: "bd279e86a2bcaa692be4e6bc618c4237"
server_key_iv: "60c1c8e3609ae32898b6f2f20c8b7590"
server_key_version: "encryptionkey@0"
I20260810 03:39:56.904462 23448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "e42b8e37147c4a63bc393030b20d2e97"
format_stamp: "Formatted at 2026-08-10 03:39:56 on dist-test-slave-ss4n"
server_key: "bd279e86a2bcaa692be4e6bc618c4237"
server_key_iv: "60c1c8e3609ae32898b6f2f20c8b7590"
server_key_version: "encryptionkey@0"
I20260810 03:39:56.908551 23448 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 03:39:56.911341 23463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:56.912736 23448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:39:56.912889 23448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "e42b8e37147c4a63bc393030b20d2e97"
format_stamp: "Formatted at 2026-08-10 03:39:56 on dist-test-slave-ss4n"
server_key: "bd279e86a2bcaa692be4e6bc618c4237"
server_key_iv: "60c1c8e3609ae32898b6f2f20c8b7590"
server_key_version: "encryptionkey@0"
I20260810 03:39:56.913072 23448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:39:56.964557 23448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:39:56.968453 23448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:39:56.968729 23448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:39:56.970362 23448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:39:56.970444 23448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:56.970520 23448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:39:56.970574 23448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:39:56 dist-test-slave-ss4n krb5kdc[23327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333196, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:39:56.982117 23448 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:43767
I20260810 03:39:56.982803 23582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:43767 every 8 connection(s)
I20260810 03:39:56.983512 23448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:39:56.986299 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:56.971351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:37237 (local address 127.21.175.126:38579)
0810 03:39:56.971484 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:56.971487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:56.972655 (+ 1168us) server_negotiation.cc:407] Connection header received
0810 03:39:56.973921 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:56.973925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:56.973999 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:56.974112 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:56.975675 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:56.976607 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:56.978230 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:56.978668 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:56.978788 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:56.982059 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:56.982063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:56.982066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:56.982109 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:56.983951 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:56.984634 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:56.984637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:56.984639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:56.984732 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:56.985105 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:56.985109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:56.985111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:56.985318 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:56.985434 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:56.985987 (+ 553us) server_negotiation.cc:299] Negotiation successful
0810 03:39:56.986126 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 03:39:56.987100 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:56.971954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38579 (local address 127.21.175.65:37237)
0810 03:39:56.972490 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:56.972535 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:56.973661 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:56.974340 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:56.974347 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:56.974810 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:39:56.975504 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:56.975528 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:56.976764 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:56.976768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:56.978085 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:56.978096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:56.978870 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:56.978874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:56.979021 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:56.979787 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:39:56.979808 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:39:56.981821 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:39:56.984128 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:56.984134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:56.984150 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:56.984500 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:56.984836 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:56.984839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:56.984841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:56.984994 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:56.985433 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:39:56.985441 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:39:56.985735 (+ 294us) client_negotiation.cc:769] Sending connection context
0810 03:39:56.985927 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 03:39:56.986272 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":144,"threads_started":1}
I20260810 03:39:56.988452 23583 heartbeater.cc:347] Connected to a master server at 127.21.175.126:38579
I20260810 03:39:56.988597 23583 heartbeater.cc:464] Registering TS with master...
I20260810 03:39:56.989099 23583 heartbeater.cc:511] Master 127.21.175.126:38579 requested a full tablet report, sending...
I20260810 03:39:56.990700 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 23448
I20260810 03:39:56.990712 23373 ts_manager.cc:194] Registered new tserver with Master: e42b8e37147c4a63bc393030b20d2e97 (127.21.175.65:43767)
I20260810 03:39:56.990845 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:39:56.991165 22205 external_mini_cluster.cc:1428] Setting key 970db4ac8896804301cecc964ba6681d
I20260810 03:39:56.992357 23373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:37237
I20260810 03:39:56.999410 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:56.991074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:35617 (local address 127.21.175.126:38579)
0810 03:39:56.991212 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:56.991216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:56.991476 (+ 260us) server_negotiation.cc:407] Connection header received
0810 03:39:56.991624 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:56.991628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:56.991675 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:56.991770 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:56.992253 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:56.992971 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:56.994199 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:56.994589 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:56.994718 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:56.995846 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:56.995850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:56.995855 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:56.995888 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:56.997594 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:56.998134 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:56.998140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:56.998143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:56.998243 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:56.998524 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:56.998530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:56.998535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:56.998771 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:56.998853 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:56.999197 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 03:39:56.999293 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 03:39:56.999643 23589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:56.990930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38579 (local address 127.21.175.65:35617)
0810 03:39:56.991373 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:56.991389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:56.991496 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:56.991796 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:56.991799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:56.992011 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:39:56.992125 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:56.992131 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:56.993117 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:56.993123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:56.994026 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:56.994037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:56.994725 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:56.994728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:56.994826 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:56.995363 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:39:56.995374 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:39:56.995712 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:39:56.997724 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:56.997727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:56.997729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:56.998034 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:56.998345 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:56.998347 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:56.998348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:56.998428 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:56.998878 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:39:56.998885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:39:56.999015 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 03:39:56.999146 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 03:39:56.999326 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":5,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:39:57.024837 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:38579
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:39:57.142287 23592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:39:57.142611 23592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:39:57.142694 23592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:39:57.146795 23592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:39:57.146903 23592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:39:57.146981 23592 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:39:57.147063 23592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:39:57.152441 23592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.175.126:38579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:39:57.153596 23592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:39:57.154673 23592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:39:57.164655 23597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:57.164639 23598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:57.164706 23600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:57.165017 23592 server_base.cc:1034] running on GCE node
I20260810 03:39:57.165525 23592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:39:57.166271 23592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:39:57.167531 23592 hybrid_clock.cc:648] HybridClock initialized: now 1786333197167503 us; error 42 us; skew 500 ppm
Aug 10 03:39:57 dist-test-slave-ss4n krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333197, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:39:57.170542 23592 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:39:57.171484 23592 webserver.cc:460] Webserver started at http://127.21.175.66:37867/ using document root <none> and password file <none>
I20260810 03:39:57.172060 23592 fs_manager.cc:359] Metadata directory not provided
I20260810 03:39:57.172171 23592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:39:57.172408 23592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:39:57.174588 23592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "93aee57ce353431193229e1ca656f72a"
format_stamp: "Formatted at 2026-08-10 03:39:57 on dist-test-slave-ss4n"
server_key: "84a2b153ad93f50ad95cc0c322048ef4"
server_key_iv: "3207130313c030f8b76fe2ee4bf2de89"
server_key_version: "encryptionkey@0"
I20260810 03:39:57.175211 23592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "93aee57ce353431193229e1ca656f72a"
format_stamp: "Formatted at 2026-08-10 03:39:57 on dist-test-slave-ss4n"
server_key: "84a2b153ad93f50ad95cc0c322048ef4"
server_key_iv: "3207130313c030f8b76fe2ee4bf2de89"
server_key_version: "encryptionkey@0"
I20260810 03:39:57.179324 23592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:39:57.181746 23607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:57.183019 23592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:39:57.183159 23592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "93aee57ce353431193229e1ca656f72a"
format_stamp: "Formatted at 2026-08-10 03:39:57 on dist-test-slave-ss4n"
server_key: "84a2b153ad93f50ad95cc0c322048ef4"
server_key_iv: "3207130313c030f8b76fe2ee4bf2de89"
server_key_version: "encryptionkey@0"
I20260810 03:39:57.183265 23592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:39:57.196789 23592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:39:57.200181 23592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:39:57.200397 23592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:39:57.202032 23592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:39:57.202109 23592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:57.202243 23592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:39:57.202296 23592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:39:57 dist-test-slave-ss4n krb5kdc[23327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333197, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:39:57.213991 23592 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:45585
I20260810 03:39:57.215174 23592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:39:57.215243 23727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:45585 every 8 connection(s)
I20260810 03:39:57.218909 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.203076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:43485 (local address 127.21.175.126:38579)
0810 03:39:57.203229 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:57.203235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:57.204417 (+ 1182us) server_negotiation.cc:407] Connection header received
0810 03:39:57.205824 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:57.205831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:57.205906 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:57.205971 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:57.208071 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.208626 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.210708 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.210947 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.211043 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.214672 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:57.214676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:57.214679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:57.214723 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:57.216710 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:57.217296 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:57.217299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:57.217300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:57.217382 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:57.217800 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:57.217802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:57.217803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:57.218058 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:57.218141 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:57.218675 (+ 534us) server_negotiation.cc:299] Negotiation successful
0810 03:39:57.218783 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 03:39:57.219743 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.203687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38579 (local address 127.21.175.66:43485)
0810 03:39:57.204214 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:57.204279 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:57.205511 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:57.206459 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:57.206472 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:57.207005 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:39:57.207817 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.207845 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.208831 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.208835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.210536 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.210552 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.211141 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.211146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.211377 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.212162 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:39:57.212184 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:39:57.214398 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:39:57.216874 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:57.216880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:57.216893 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:57.217192 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:57.217535 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:57.217541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:57.217544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:57.217699 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:57.218155 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:39:57.218160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:39:57.218407 (+ 247us) client_negotiation.cc:769] Sending connection context
0810 03:39:57.218566 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 03:39:57.218827 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":121,"threads_started":1}
I20260810 03:39:57.221285 23728 heartbeater.cc:347] Connected to a master server at 127.21.175.126:38579
I20260810 03:39:57.221431 23728 heartbeater.cc:464] Registering TS with master...
I20260810 03:39:57.222018 23728 heartbeater.cc:511] Master 127.21.175.126:38579 requested a full tablet report, sending...
I20260810 03:39:57.222733 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 23592
I20260810 03:39:57.222836 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:39:57.223112 22205 external_mini_cluster.cc:1428] Setting key ae889b7987b9df20f376eae9082ea4de
I20260810 03:39:57.223174 23373 ts_manager.cc:194] Registered new tserver with Master: 93aee57ce353431193229e1ca656f72a (127.21.175.66:45585)
I20260810 03:39:57.223933 23373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:43485
I20260810 03:39:57.231571 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.223647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38579 (local address 127.21.175.66:60055)
0810 03:39:57.223970 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:57.223987 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:57.224090 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:57.224345 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:57.224348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:57.224584 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:39:57.224727 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.224736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.225801 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.225804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.226517 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.226527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.227006 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.227010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.227118 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.227690 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:39:57.227702 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:39:57.228027 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:39:57.229963 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:57.229968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:57.229970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:57.230190 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:57.230481 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:57.230485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:57.230487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:57.230575 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:57.231001 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:39:57.231004 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:39:57.231115 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 03:39:57.231226 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 03:39:57.231387 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":114,"threads_started":1}
I20260810 03:39:57.231560 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.223933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:60055 (local address 127.21.175.126:38579)
0810 03:39:57.224046 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:57.224050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:57.224065 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:39:57.224195 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:57.224199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:57.224246 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:57.224319 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:57.224858 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.225657 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.226642 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.226881 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.226989 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.228190 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:57.228193 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:57.228196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:57.228230 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:57.229811 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:57.230296 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:57.230300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:57.230302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:57.230385 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:57.230669 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:57.230674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:57.230675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:57.230898 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:57.231011 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:57.231284 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 03:39:57.231396 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:39:57.252914 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:38579
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:39:57.371129 23739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:39:57.371417 23739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:39:57.371495 23739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:39:57.375819 23739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:39:57.375912 23739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:39:57.375986 23739 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:39:57.376065 23739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:39:57.381705 23739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.175.126:38579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:39:57.382908 23739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:39:57.384040 23739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:39:57.395112 23750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:57.395268 23748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:57.395170 23739 server_base.cc:1034] running on GCE node
W20260810 03:39:57.395176 23747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:57.395866 23739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:39:57.396484 23739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:39:57.397691 23739 hybrid_clock.cc:648] HybridClock initialized: now 1786333197397653 us; error 46 us; skew 500 ppm
Aug 10 03:39:57 dist-test-slave-ss4n krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333197, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:39:57.401121 23739 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:39:57.402278 23739 webserver.cc:460] Webserver started at http://127.21.175.67:46407/ using document root <none> and password file <none>
I20260810 03:39:57.402873 23739 fs_manager.cc:359] Metadata directory not provided
I20260810 03:39:57.402956 23739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:39:57.403152 23739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:39:57.405473 23739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "09bcadd1a0484f94adb825885ee53688"
format_stamp: "Formatted at 2026-08-10 03:39:57 on dist-test-slave-ss4n"
server_key: "a403e677b70bb17bb7f07e35fc0356f2"
server_key_iv: "d385873b5f889b6c2007911aec70e2e3"
server_key_version: "encryptionkey@0"
I20260810 03:39:57.406208 23739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "09bcadd1a0484f94adb825885ee53688"
format_stamp: "Formatted at 2026-08-10 03:39:57 on dist-test-slave-ss4n"
server_key: "a403e677b70bb17bb7f07e35fc0356f2"
server_key_iv: "d385873b5f889b6c2007911aec70e2e3"
server_key_version: "encryptionkey@0"
I20260810 03:39:57.410569 23739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:39:57.413609 23757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:57.415073 23739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:39:57.415221 23739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "09bcadd1a0484f94adb825885ee53688"
format_stamp: "Formatted at 2026-08-10 03:39:57 on dist-test-slave-ss4n"
server_key: "a403e677b70bb17bb7f07e35fc0356f2"
server_key_iv: "d385873b5f889b6c2007911aec70e2e3"
server_key_version: "encryptionkey@0"
I20260810 03:39:57.415341 23739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:39:57.427207 23739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:39:57.430680 23739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:39:57.430895 23739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:39:57.432440 23739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:39:57.432514 23739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:57.432590 23739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:39:57.432641 23739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:39:57 dist-test-slave-ss4n krb5kdc[23327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333197, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:39:57.444044 23739 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:40129
I20260810 03:39:57.444178 23881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:40129 every 8 connection(s)
I20260810 03:39:57.445370 23739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:39:57.447606 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.433283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:59591 (local address 127.21.175.126:38579)
0810 03:39:57.433421 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:57.433424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:57.434546 (+ 1122us) server_negotiation.cc:407] Connection header received
0810 03:39:57.435782 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:57.435787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:57.435863 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:57.435979 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:57.437372 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.438033 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.439335 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.439645 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.439768 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.443460 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:57.443464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:57.443466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:57.443495 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:57.445222 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:57.445870 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:57.445877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:57.445881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:57.445992 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:57.446407 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:57.446410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:57.446412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:57.446633 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:57.446749 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:57.447281 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 03:39:57.447419 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:39:57.448452 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.433890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38579 (local address 127.21.175.67:59591)
0810 03:39:57.434387 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:57.434431 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:57.435433 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:57.436220 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:57.436230 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:57.436665 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:39:57.437197 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.437219 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.438177 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.438181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.439198 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.439209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.439823 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.439827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.440038 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.440722 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:39:57.440740 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:39:57.443255 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:39:57.445359 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:57.445364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:57.445377 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:57.445755 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:57.446119 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:57.446122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:57.446124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:57.446292 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:57.446748 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:39:57.446757 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:39:57.447011 (+ 254us) client_negotiation.cc:769] Sending connection context
0810 03:39:57.447217 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 03:39:57.447552 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":145,"threads_started":1}
I20260810 03:39:57.449819 23882 heartbeater.cc:347] Connected to a master server at 127.21.175.126:38579
I20260810 03:39:57.449988 23882 heartbeater.cc:464] Registering TS with master...
I20260810 03:39:57.450575 23882 heartbeater.cc:511] Master 127.21.175.126:38579 requested a full tablet report, sending...
I20260810 03:39:57.450973 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 23739
I20260810 03:39:57.451107 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:39:57.451395 22205 external_mini_cluster.cc:1428] Setting key 8e29cc5d9d219b519dda541fd6297cd8
I20260810 03:39:57.451678 23373 ts_manager.cc:194] Registered new tserver with Master: 09bcadd1a0484f94adb825885ee53688 (127.21.175.67:40129)
I20260810 03:39:57.452486 23373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:59591
I20260810 03:39:57.453734 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:39:57.460960 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.452334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38579 (local address 127.21.175.67:48891)
0810 03:39:57.452906 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:57.452922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:57.453035 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:57.453279 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:57.453282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:57.453542 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:39:57.453654 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.453665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.454707 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.454711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.455274 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.455282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.455934 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.455939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.456109 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.456664 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:39:57.456684 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:39:57.457193 (+ 509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:39:57.459058 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:57.459064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:57.459069 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:57.459395 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:57.459808 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:57.459813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:57.459815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:57.459919 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:57.460472 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:39:57.460475 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:39:57.460576 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 03:39:57.460664 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 03:39:57.460792 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":107,"threads_started":1}
I20260810 03:39:57.461153 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.452150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:48891 (local address 127.21.175.126:38579)
0810 03:39:57.452251 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:57.452255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:57.453013 (+ 758us) server_negotiation.cc:407] Connection header received
0810 03:39:57.453114 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:57.453117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:57.453161 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:57.453242 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:57.453766 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.454556 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.455392 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.455786 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.455903 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.457347 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:57.457350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:57.457352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:57.457376 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:57.458922 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:57.459601 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:57.459604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:57.459606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:57.459702 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:57.460057 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:57.460060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:57.460061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:57.460256 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:57.460483 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:57.460766 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 03:39:57.460905 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Aug 10 03:39:57 dist-test-slave-ss4n krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333197, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:39:57 dist-test-slave-ss4n krb5kdc[23327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333197, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:39:57.485101 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.475730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42852 (local address 127.21.175.126:38579)
0810 03:39:57.475932 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:57.475936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:57.475954 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:39:57.476024 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:57.476027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:57.476075 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:57.476224 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:57.477010 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.477758 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.478720 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.478978 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.479086 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.482093 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:57.482096 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:57.482098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:57.482132 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:57.483537 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:57.483983 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:57.483985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:57.483987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:57.484065 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:57.484319 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:57.484321 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:57.484322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:57.484510 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:57.484659 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:57.484865 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 03:39:57.484973 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260810 03:39:57.488765 23374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:39:57.492100 23374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:39:57.507776 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.502188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56982 (local address 127.21.175.66:45585)
0810 03:39:57.502635 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:57.502650 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:57.502673 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:39:57.502807 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:57.502812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:57.503022 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:57.503177 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:39:57.503495 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.504228 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.506476 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.507051 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.507178 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.507366 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:57.507515 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 03:39:57.507613 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":113,"threads_started":1}
I20260810 03:39:57.507767 23904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.501742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:45585 (local address 127.0.0.1:56982)
0810 03:39:57.502591 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:57.502604 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:57.502689 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:57.503290 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:57.503293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:57.503307 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:39:57.503375 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.503379 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.504352 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.504357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.506337 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.506344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.507180 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.507184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.507299 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.507339 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 03:39:57.507477 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 03:39:57.507613 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":307,"thread_start_us":88,"threads_started":1}
I20260810 03:39:57.507942 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.501341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:43767 (local address 127.0.0.1:59526)
0810 03:39:57.502276 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:57.502293 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:57.502386 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:57.503072 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:57.503075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:57.503095 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:39:57.503191 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.503199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.505108 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.505112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.506211 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.506222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.507421 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.507425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.507521 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.507554 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 03:39:57.507631 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 03:39:57.507710 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":65,"thread_start_us":97,"threads_started":1}
I20260810 03:39:57.508608 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.501825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.21.175.65:43767)
0810 03:39:57.502550 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:57.502557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:57.502593 (+ 36us) server_negotiation.cc:407] Connection header received
0810 03:39:57.502675 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:57.502683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:57.502914 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:57.503350 (+ 436us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:39:57.503489 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.504965 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.506341 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.507260 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.508119 (+ 859us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.508276 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:57.508332 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 03:39:57.508430 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":95,"threads_started":1}
I20260810 03:39:57.508741 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.500946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:40129 (local address 127.0.0.1:45928)
0810 03:39:57.501354 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:57.501412 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:57.501609 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:57.502738 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:57.502745 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:57.502771 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:39:57.502912 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.502921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.504504 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.504507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.506316 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.506329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.507636 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.507640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.508407 (+ 767us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.508430 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 03:39:57.508509 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 03:39:57.508581 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":229,"thread_start_us":96,"threads_started":1}
I20260810 03:39:57.509133 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.501112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.21.175.67:40129)
0810 03:39:57.501946 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:57.501966 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:57.501993 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:39:57.502091 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:57.502100 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:57.502366 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:57.502543 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:39:57.503069 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.504382 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.506449 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.507260 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.507394 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.507567 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:57.508918 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0810 03:39:57.509005 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":123,"threads_started":1}
I20260810 03:39:57.511339 23515 tablet_service.cc:1467] Processing CreateTablet for tablet 0382fbf4b64f4bbd84053c77d2096730 (DEFAULT_TABLE table=test-table [id=207f2ee620ac4547801fabff98ad1f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:39:57.511572 23657 tablet_service.cc:1467] Processing CreateTablet for tablet 0382fbf4b64f4bbd84053c77d2096730 (DEFAULT_TABLE table=test-table [id=207f2ee620ac4547801fabff98ad1f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:39:57.512496 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0382fbf4b64f4bbd84053c77d2096730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:39:57.512064 23812 tablet_service.cc:1467] Processing CreateTablet for tablet 0382fbf4b64f4bbd84053c77d2096730 (DEFAULT_TABLE table=test-table [id=207f2ee620ac4547801fabff98ad1f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:39:57.512689 23657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0382fbf4b64f4bbd84053c77d2096730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:39:57.513116 23812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0382fbf4b64f4bbd84053c77d2096730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:39:57.519027 23907 tablet_bootstrap.cc:492] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97: Bootstrap starting.
I20260810 03:39:57.521644 23909 tablet_bootstrap.cc:492] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a: Bootstrap starting.
I20260810 03:39:57.521709 23907 tablet_bootstrap.cc:654] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97: Neither blocks nor log segments found. Creating new log.
I20260810 03:39:57.522269 23908 tablet_bootstrap.cc:492] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688: Bootstrap starting.
I20260810 03:39:57.522921 23907 log.cc:824] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97: Log is configured to *not* fsync() on all Append() calls
I20260810 03:39:57.524681 23908 tablet_bootstrap.cc:654] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688: Neither blocks nor log segments found. Creating new log.
I20260810 03:39:57.525040 23907 tablet_bootstrap.cc:492] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97: No bootstrap required, opened a new log
I20260810 03:39:57.525288 23907 ts_tablet_manager.cc:1397] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 03:39:57.525573 23908 log.cc:824] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688: Log is configured to *not* fsync() on all Append() calls
I20260810 03:39:57.525955 23909 tablet_bootstrap.cc:654] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a: Neither blocks nor log segments found. Creating new log.
I20260810 03:39:57.526729 23909 log.cc:824] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a: Log is configured to *not* fsync() on all Append() calls
I20260810 03:39:57.527843 23908 tablet_bootstrap.cc:492] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688: No bootstrap required, opened a new log
I20260810 03:39:57.528050 23908 ts_tablet_manager.cc:1397] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:39:57.528970 23909 tablet_bootstrap.cc:492] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a: No bootstrap required, opened a new log
I20260810 03:39:57.529167 23909 ts_tablet_manager.cc:1397] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260810 03:39:57.530273 23907 raft_consensus.cc:348] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } }
I20260810 03:39:57.530614 23907 raft_consensus.cc:374] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:39:57.530686 23907 raft_consensus.cc:729] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e42b8e37147c4a63bc393030b20d2e97, State: Initialized, Role: FOLLOWER
I20260810 03:39:57.531231 23907 consensus_queue.cc:260] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } }
I20260810 03:39:57.531961 23583 heartbeater.cc:503] Master 127.21.175.126:38579 was elected leader, sending a full tablet report...
I20260810 03:39:57.532395 23907 ts_tablet_manager.cc:1428] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 03:39:57.533193 23909 raft_consensus.cc:348] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } }
I20260810 03:39:57.533506 23909 raft_consensus.cc:374] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:39:57.533360 23908 raft_consensus.cc:348] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } }
I20260810 03:39:57.533568 23909 raft_consensus.cc:729] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93aee57ce353431193229e1ca656f72a, State: Initialized, Role: FOLLOWER
I20260810 03:39:57.533614 23908 raft_consensus.cc:374] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:39:57.533674 23908 raft_consensus.cc:729] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bcadd1a0484f94adb825885ee53688, State: Initialized, Role: FOLLOWER
I20260810 03:39:57.534035 23909 consensus_queue.cc:260] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } }
I20260810 03:39:57.534049 23908 consensus_queue.cc:260] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } }
I20260810 03:39:57.534783 23882 heartbeater.cc:503] Master 127.21.175.126:38579 was elected leader, sending a full tablet report...
I20260810 03:39:57.535151 23728 heartbeater.cc:503] Master 127.21.175.126:38579 was elected leader, sending a full tablet report...
I20260810 03:39:57.536128 23909 ts_tablet_manager.cc:1428] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 03:39:57.536690 23908 ts_tablet_manager.cc:1428] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260810 03:39:57.550433 23916 raft_consensus.cc:482] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:39:57.550683 23916 raft_consensus.cc:504] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } }
I20260810 03:39:57.551976 23916 leader_election.cc:288] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09bcadd1a0484f94adb825885ee53688 (127.21.175.67:40129), e42b8e37147c4a63bc393030b20d2e97 (127.21.175.65:43767)
I20260810 03:39:57.556555 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.552391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:38711 (local address 127.21.175.65:43767)
0810 03:39:57.552547 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:57.552551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:57.552749 (+ 198us) server_negotiation.cc:407] Connection header received
0810 03:39:57.552854 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:57.552857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:57.552908 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:57.552998 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:39:57.553270 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.554115 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.555426 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.555911 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.556018 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.556064 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:57.556372 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 03:39:57.556439 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:39:57.556610 23917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.552277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:43767 (local address 127.21.175.66:38711)
0810 03:39:57.552653 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:57.552664 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:57.552755 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:57.553022 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:57.553024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:57.553039 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:39:57.553163 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.553169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.554245 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.554249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.555292 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.555303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.556053 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.556057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.556209 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.556249 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 03:39:57.556348 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 03:39:57.556445 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":21,"thread_start_us":143,"threads_started":1}
I20260810 03:39:57.557289 23536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0382fbf4b64f4bbd84053c77d2096730" candidate_uuid: "93aee57ce353431193229e1ca656f72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e42b8e37147c4a63bc393030b20d2e97" is_pre_election: true
I20260810 03:39:57.557735 23536 raft_consensus.cc:2454] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93aee57ce353431193229e1ca656f72a in term 0.
I20260810 03:39:57.558291 23609 leader_election.cc:302] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93aee57ce353431193229e1ca656f72a, e42b8e37147c4a63bc393030b20d2e97; no voters:
I20260810 03:39:57.558755 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.552196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:40129 (local address 127.21.175.66:59053)
0810 03:39:57.552381 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:57.552397 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:57.552496 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:57.552839 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:57.552844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:57.552863 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:39:57.553025 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.553032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.554245 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.554249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.557424 (+ 3175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.557436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.558310 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.558316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.558428 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.558453 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:39:57.558534 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 03:39:57.558609 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":38}
I20260810 03:39:57.558704 23916 raft_consensus.cc:2781] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:39:57.558820 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.552296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:59053 (local address 127.21.175.67:40129)
0810 03:39:57.552413 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:57.552420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:57.552449 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:39:57.552651 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:57.552655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:57.552703 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:57.552817 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:39:57.553166 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.554110 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.557592 (+ 3482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.558191 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.558296 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.558339 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:57.558576 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 03:39:57.558677 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 03:39:57.558857 23916 raft_consensus.cc:482] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:39:57.558925 23916 raft_consensus.cc:3037] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:39:57.559414 23834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0382fbf4b64f4bbd84053c77d2096730" candidate_uuid: "93aee57ce353431193229e1ca656f72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09bcadd1a0484f94adb825885ee53688" is_pre_election: true
I20260810 03:39:57.559702 23834 raft_consensus.cc:2454] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93aee57ce353431193229e1ca656f72a in term 0.
I20260810 03:39:57.560415 23916 raft_consensus.cc:504] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } }
I20260810 03:39:57.560778 23916 leader_election.cc:288] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [CANDIDATE]: Term 1 election: Requested vote from peers 09bcadd1a0484f94adb825885ee53688 (127.21.175.67:40129), e42b8e37147c4a63bc393030b20d2e97 (127.21.175.65:43767)
I20260810 03:39:57.561124 23834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0382fbf4b64f4bbd84053c77d2096730" candidate_uuid: "93aee57ce353431193229e1ca656f72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09bcadd1a0484f94adb825885ee53688"
I20260810 03:39:57.561235 23536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0382fbf4b64f4bbd84053c77d2096730" candidate_uuid: "93aee57ce353431193229e1ca656f72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e42b8e37147c4a63bc393030b20d2e97"
I20260810 03:39:57.561275 23834 raft_consensus.cc:3037] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:39:57.561374 23536 raft_consensus.cc:3037] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:39:57.562862 23536 raft_consensus.cc:2454] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93aee57ce353431193229e1ca656f72a in term 1.
I20260810 03:39:57.562862 23834 raft_consensus.cc:2454] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93aee57ce353431193229e1ca656f72a in term 1.
I20260810 03:39:57.563285 23610 leader_election.cc:302] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09bcadd1a0484f94adb825885ee53688, 93aee57ce353431193229e1ca656f72a; no voters:
I20260810 03:39:57.563506 23916 raft_consensus.cc:2781] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:39:57.563758 23916 raft_consensus.cc:686] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [term 1 LEADER]: Becoming Leader. State: Replica: 93aee57ce353431193229e1ca656f72a, State: Running, Role: LEADER
I20260810 03:39:57.564198 23916 consensus_queue.cc:237] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } }
I20260810 03:39:57.567615 23374 catalog_manager.cc:5495] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a reported cstate change: term changed from 0 to 1, leader changed from <none> to 93aee57ce353431193229e1ca656f72a (127.21.175.66). New cstate: current_term: 1 leader_uuid: "93aee57ce353431193229e1ca656f72a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } health_report { overall_health: UNKNOWN } } }
I20260810 03:39:57.595440 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.591340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56996 (local address 127.21.175.66:45585)
0810 03:39:57.591498 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:57.591502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:57.591563 (+ 61us) server_negotiation.cc:407] Connection header received
0810 03:39:57.591795 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:57.591798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:57.591862 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:57.591959 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:39:57.592251 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.592796 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.593850 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.594089 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.594175 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.594569 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:39:57.595017 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:39:57.595126 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:57.595254 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 03:39:57.595317 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 03:39:57.603251 23834 raft_consensus.cc:1261] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688 [term 1 FOLLOWER]: Refusing update from remote peer 93aee57ce353431193229e1ca656f72a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:39:57.604089 23536 raft_consensus.cc:1261] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97 [term 1 FOLLOWER]: Refusing update from remote peer 93aee57ce353431193229e1ca656f72a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:39:57.604269 23916 consensus_queue.cc:1035] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:39:57.604748 23918 consensus_queue.cc:1035] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:39:57.612258 23922 mvcc.cc:205] Tried to move back new op lower bound from 7316820777369030656 to 7316820777225945088. Current Snapshot: MvccSnapshot[applied={T|T < 7316820777369030656}]
I20260810 03:39:57.627058 23373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:39:57.627374 23373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:39:57.630992 23373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b: Sending DeleteTablet for 3 replicas of tablet 0382fbf4b64f4bbd84053c77d2096730
I20260810 03:39:57.632085 23515 tablet_service.cc:1514] Processing DeleteTablet for tablet 0382fbf4b64f4bbd84053c77d2096730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:39:57 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:59526
I20260810 03:39:57.632292 23812 tablet_service.cc:1514] Processing DeleteTablet for tablet 0382fbf4b64f4bbd84053c77d2096730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:39:57 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:45928
I20260810 03:39:57.632310 23657 tablet_service.cc:1514] Processing DeleteTablet for tablet 0382fbf4b64f4bbd84053c77d2096730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:39:57 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:56982
I20260810 03:39:57.633136 23932 tablet_replica.cc:330] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a: stopping tablet replica
I20260810 03:39:57.633188 23934 tablet_replica.cc:330] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97: stopping tablet replica
I20260810 03:39:57.633575 23932 raft_consensus.cc:2229] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:39:57.633575 23934 raft_consensus.cc:2229] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:39:57.633845 23373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 03:39:57.633963 23934 raft_consensus.cc:2258] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:39:57.634258 23932 raft_consensus.cc:2258] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 03:39:57.634341 23373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:39:57.635525 23933 tablet_replica.cc:330] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688: stopping tablet replica
I20260810 03:39:57.635875 23933 raft_consensus.cc:2229] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:39:57.635897 23934 ts_tablet_manager.cc:1905] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:39:57.636236 23933 raft_consensus.cc:2258] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:39:57.638096 23932 ts_tablet_manager.cc:1905] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:39:57.639503 23933 ts_tablet_manager.cc:1905] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:39:57.639503 23934 ts_tablet_manager.cc:1918] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:39:57.639626 23934 log.cc:1196] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/wal/wals/0382fbf4b64f4bbd84053c77d2096730
I20260810 03:39:57.640033 23934 ts_tablet_manager.cc:1939] T 0382fbf4b64f4bbd84053c77d2096730 P e42b8e37147c4a63bc393030b20d2e97: Deleting consensus metadata
I20260810 03:39:57.641474 23358 catalog_manager.cc:4841] TS e42b8e37147c4a63bc393030b20d2e97 (127.21.175.65:43767): tablet 0382fbf4b64f4bbd84053c77d2096730 (table test-table [id=207f2ee620ac4547801fabff98ad1f6a]) successfully deleted
I20260810 03:39:57.641563 23515 tablet_service.cc:1467] Processing CreateTablet for tablet c0b361d0328047dab36863ad3301c748 (DEFAULT_TABLE table=test-table [id=af40c7bc80314cf3bbd71939dbead758]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:39:57.641852 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b361d0328047dab36863ad3301c748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:39:57.643910 23907 tablet_bootstrap.cc:492] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97: Bootstrap starting.
I20260810 03:39:57.644163 23657 tablet_service.cc:1467] Processing CreateTablet for tablet c0b361d0328047dab36863ad3301c748 (DEFAULT_TABLE table=test-table [id=af40c7bc80314cf3bbd71939dbead758]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:39:57.644325 23933 ts_tablet_manager.cc:1918] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:39:57.644429 23933 log.cc:1196] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/0382fbf4b64f4bbd84053c77d2096730
I20260810 03:39:57.644480 23657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b361d0328047dab36863ad3301c748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:39:57.644775 23933 ts_tablet_manager.cc:1939] T 0382fbf4b64f4bbd84053c77d2096730 P 09bcadd1a0484f94adb825885ee53688: Deleting consensus metadata
I20260810 03:39:57.645056 23907 tablet_bootstrap.cc:654] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97: Neither blocks nor log segments found. Creating new log.
I20260810 03:39:57.645799 23812 tablet_service.cc:1467] Processing CreateTablet for tablet c0b361d0328047dab36863ad3301c748 (DEFAULT_TABLE table=test-table [id=af40c7bc80314cf3bbd71939dbead758]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:39:57.646091 23812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b361d0328047dab36863ad3301c748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:39:57.646409 23907 tablet_bootstrap.cc:492] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97: No bootstrap required, opened a new log
I20260810 03:39:57.646499 23907 ts_tablet_manager.cc:1397] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:39:57.647009 23907 raft_consensus.cc:348] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:57.647146 23907 raft_consensus.cc:374] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:39:57.647185 23907 raft_consensus.cc:729] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e42b8e37147c4a63bc393030b20d2e97, State: Initialized, Role: FOLLOWER
I20260810 03:39:57.647344 23907 consensus_queue.cc:260] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:57.647725 23907 ts_tablet_manager.cc:1428] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 03:39:57.649459 23908 tablet_bootstrap.cc:492] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688: Bootstrap starting.
I20260810 03:39:57.650691 23908 tablet_bootstrap.cc:654] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688: Neither blocks nor log segments found. Creating new log.
I20260810 03:39:57.651232 23359 catalog_manager.cc:4841] TS 09bcadd1a0484f94adb825885ee53688 (127.21.175.67:40129): tablet 0382fbf4b64f4bbd84053c77d2096730 (table test-table [id=207f2ee620ac4547801fabff98ad1f6a]) successfully deleted
I20260810 03:39:57.653064 23908 tablet_bootstrap.cc:492] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688: No bootstrap required, opened a new log
I20260810 03:39:57.653151 23908 ts_tablet_manager.cc:1397] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260810 03:39:57.653690 23908 raft_consensus.cc:348] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:57.653813 23908 raft_consensus.cc:374] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:39:57.653846 23908 raft_consensus.cc:729] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bcadd1a0484f94adb825885ee53688, State: Initialized, Role: FOLLOWER
I20260810 03:39:57.654006 23908 consensus_queue.cc:260] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:57.654285 23908 ts_tablet_manager.cc:1428] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 03:39:57.656488 23909 tablet_bootstrap.cc:492] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a: Bootstrap starting.
I20260810 03:39:57.658905 23932 ts_tablet_manager.cc:1918] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:39:57.659106 23932 log.cc:1196] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/0382fbf4b64f4bbd84053c77d2096730
I20260810 03:39:57.659483 23932 ts_tablet_manager.cc:1939] T 0382fbf4b64f4bbd84053c77d2096730 P 93aee57ce353431193229e1ca656f72a: Deleting consensus metadata
I20260810 03:39:57.659701 23909 tablet_bootstrap.cc:654] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a: Neither blocks nor log segments found. Creating new log.
I20260810 03:39:57.660904 23359 catalog_manager.cc:4841] TS 93aee57ce353431193229e1ca656f72a (127.21.175.66:45585): tablet 0382fbf4b64f4bbd84053c77d2096730 (table test-table [id=207f2ee620ac4547801fabff98ad1f6a]) successfully deleted
I20260810 03:39:57.661563 23909 tablet_bootstrap.cc:492] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a: No bootstrap required, opened a new log
I20260810 03:39:57.661674 23909 ts_tablet_manager.cc:1397] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 03:39:57.662251 23909 raft_consensus.cc:348] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:57.662405 23909 raft_consensus.cc:374] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:39:57.662448 23909 raft_consensus.cc:729] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93aee57ce353431193229e1ca656f72a, State: Initialized, Role: FOLLOWER
I20260810 03:39:57.662588 23909 consensus_queue.cc:260] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:57.662909 23909 ts_tablet_manager.cc:1428] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260810 03:39:57.696512 23883 tablet.cc:2369] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:39:57.716732 23729 tablet.cc:2369] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:39:57.734975 23584 tablet.cc:2369] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:39:57.925624 23928 raft_consensus.cc:482] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:39:57.925802 23928 raft_consensus.cc:504] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:57.926352 23928 leader_election.cc:288] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e42b8e37147c4a63bc393030b20d2e97 (127.21.175.65:43767), 09bcadd1a0484f94adb825885ee53688 (127.21.175.67:40129)
I20260810 03:39:57.926868 23536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b361d0328047dab36863ad3301c748" candidate_uuid: "93aee57ce353431193229e1ca656f72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e42b8e37147c4a63bc393030b20d2e97" is_pre_election: true
I20260810 03:39:57.926894 23834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b361d0328047dab36863ad3301c748" candidate_uuid: "93aee57ce353431193229e1ca656f72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09bcadd1a0484f94adb825885ee53688" is_pre_election: true
I20260810 03:39:57.927055 23536 raft_consensus.cc:2454] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93aee57ce353431193229e1ca656f72a in term 0.
I20260810 03:39:57.927067 23834 raft_consensus.cc:2454] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93aee57ce353431193229e1ca656f72a in term 0.
I20260810 03:39:57.927395 23609 leader_election.cc:302] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93aee57ce353431193229e1ca656f72a, e42b8e37147c4a63bc393030b20d2e97; no voters:
I20260810 03:39:57.927603 23928 raft_consensus.cc:2781] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:39:57.927662 23928 raft_consensus.cc:482] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:39:57.927690 23928 raft_consensus.cc:3037] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:39:57.928721 23928 raft_consensus.cc:504] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:57.929075 23928 leader_election.cc:288] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [CANDIDATE]: Term 1 election: Requested vote from peers e42b8e37147c4a63bc393030b20d2e97 (127.21.175.65:43767), 09bcadd1a0484f94adb825885ee53688 (127.21.175.67:40129)
I20260810 03:39:57.929523 23834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b361d0328047dab36863ad3301c748" candidate_uuid: "93aee57ce353431193229e1ca656f72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09bcadd1a0484f94adb825885ee53688"
I20260810 03:39:57.929461 23536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b361d0328047dab36863ad3301c748" candidate_uuid: "93aee57ce353431193229e1ca656f72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e42b8e37147c4a63bc393030b20d2e97"
I20260810 03:39:57.929661 23834 raft_consensus.cc:3037] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:39:57.929687 23536 raft_consensus.cc:3037] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:39:57.930603 23536 raft_consensus.cc:2454] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93aee57ce353431193229e1ca656f72a in term 1.
I20260810 03:39:57.930650 23834 raft_consensus.cc:2454] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93aee57ce353431193229e1ca656f72a in term 1.
I20260810 03:39:57.930912 23609 leader_election.cc:302] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93aee57ce353431193229e1ca656f72a, e42b8e37147c4a63bc393030b20d2e97; no voters:
I20260810 03:39:57.931106 23928 raft_consensus.cc:2781] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:39:57.931203 23928 raft_consensus.cc:686] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [term 1 LEADER]: Becoming Leader. State: Replica: 93aee57ce353431193229e1ca656f72a, State: Running, Role: LEADER
I20260810 03:39:57.931370 23928 consensus_queue.cc:237] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:57.933200 23373 catalog_manager.cc:5495] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a reported cstate change: term changed from 0 to 1, leader changed from <none> to 93aee57ce353431193229e1ca656f72a (127.21.175.66). New cstate: current_term: 1 leader_uuid: "93aee57ce353431193229e1ca656f72a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } health_report { overall_health: UNKNOWN } } }
Aug 10 03:39:57 dist-test-slave-ss4n krb5kdc[23327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333196, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:39:57.980778 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.971494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42858 (local address 127.21.175.126:38579)
0810 03:39:57.971681 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:57.971685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:57.971880 (+ 195us) server_negotiation.cc:407] Connection header received
0810 03:39:57.972050 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:57.972054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:57.972100 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:57.972197 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:57.973002 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.973770 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.974726 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.975076 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.975178 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.977844 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:57.977848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:57.977850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:57.977872 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:57.979166 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:57.979639 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:57.979642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:57.979643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:57.979716 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:57.979957 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:57.979959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:57.979960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:57.980149 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:57.980225 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:57.980504 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 03:39:57.980632 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":47}
I20260810 03:39:57.980786 23944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:57.971408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38579 (local address 127.0.0.1:42858)
0810 03:39:57.971787 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:57.971799 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:57.971940 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:57.972230 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:57.972232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:57.972741 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:39:57.972835 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.972841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.973892 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.973894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.974572 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:57.974583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:57.975199 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:57.975201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:57.975298 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:57.976011 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:39:57.976024 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:39:57.977704 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:39:57.979268 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:57.979278 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:57.979284 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:57.979527 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:57.979796 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:39:57.979798 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:39:57.979799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:39:57.979880 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:39:57.980240 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:39:57.980246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:39:57.980351 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 03:39:57.980445 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 03:39:57.980573 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":109,"threads_started":1}
I20260810 03:39:57.985934 23373 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:42858:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 03:39:57.986924 23373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:39:57.992322 23812 tablet_service.cc:1467] Processing CreateTablet for tablet 0ea80a217b4b4ae6972108559ed3caa5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1b738c41729d4adb9615825be0e219a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:39:57.992293 23515 tablet_service.cc:1467] Processing CreateTablet for tablet 0ea80a217b4b4ae6972108559ed3caa5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1b738c41729d4adb9615825be0e219a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:39:57.992332 23657 tablet_service.cc:1467] Processing CreateTablet for tablet 0ea80a217b4b4ae6972108559ed3caa5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1b738c41729d4adb9615825be0e219a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:39:57.992722 23812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ea80a217b4b4ae6972108559ed3caa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:39:57.992720 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ea80a217b4b4ae6972108559ed3caa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:39:57.992750 23657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ea80a217b4b4ae6972108559ed3caa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:39:57.995500 23908 tablet_bootstrap.cc:492] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688: Bootstrap starting.
I20260810 03:39:57.995582 23909 tablet_bootstrap.cc:492] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a: Bootstrap starting.
I20260810 03:39:57.995697 23907 tablet_bootstrap.cc:492] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97: Bootstrap starting.
I20260810 03:39:57.996480 23908 tablet_bootstrap.cc:654] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688: Neither blocks nor log segments found. Creating new log.
I20260810 03:39:57.996521 23907 tablet_bootstrap.cc:654] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97: Neither blocks nor log segments found. Creating new log.
I20260810 03:39:57.996521 23909 tablet_bootstrap.cc:654] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a: Neither blocks nor log segments found. Creating new log.
I20260810 03:39:57.998303 23909 tablet_bootstrap.cc:492] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a: No bootstrap required, opened a new log
I20260810 03:39:57.998385 23909 ts_tablet_manager.cc:1397] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:39:57.998733 23907 tablet_bootstrap.cc:492] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97: No bootstrap required, opened a new log
I20260810 03:39:57.998793 23907 ts_tablet_manager.cc:1397] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 03:39:57.998777 23909 raft_consensus.cc:348] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:57.998881 23909 raft_consensus.cc:374] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:39:57.998909 23909 raft_consensus.cc:729] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93aee57ce353431193229e1ca656f72a, State: Initialized, Role: FOLLOWER
I20260810 03:39:57.999032 23909 consensus_queue.cc:260] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:57.999287 23907 raft_consensus.cc:348] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:57.999285 23928 tablet_replica.cc:439] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } }
I20260810 03:39:57.999408 23907 raft_consensus.cc:374] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:39:57.999433 23928 tablet_replica.cc:442] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:39:57.999444 23907 raft_consensus.cc:729] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e42b8e37147c4a63bc393030b20d2e97, State: Initialized, Role: FOLLOWER
I20260810 03:39:57.999563 23907 consensus_queue.cc:260] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:57.999298 23909 ts_tablet_manager.cc:1428] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:39:57.999908 23907 ts_tablet_manager.cc:1428] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 03:39:58.000058 23913 tablet_replica.cc:439] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } }
I20260810 03:39:58.000198 23913 tablet_replica.cc:442] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:39:58.005620 23908 tablet_bootstrap.cc:492] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688: No bootstrap required, opened a new log
I20260810 03:39:58.005721 23908 ts_tablet_manager.cc:1397] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260810 03:39:58.006086 23908 raft_consensus.cc:348] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:58.006189 23908 raft_consensus.cc:374] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:39:58.006211 23908 raft_consensus.cc:729] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bcadd1a0484f94adb825885ee53688, State: Initialized, Role: FOLLOWER
I20260810 03:39:58.006307 23908 consensus_queue.cc:260] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:58.006639 23908 ts_tablet_manager.cc:1428] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 03:39:58.006537 23915 tablet_replica.cc:439] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } }
I20260810 03:39:58.006755 23915 tablet_replica.cc:442] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:39:58.150511 23915 raft_consensus.cc:482] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:39:58.150691 23915 raft_consensus.cc:504] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:58.152068 23915 leader_election.cc:288] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e42b8e37147c4a63bc393030b20d2e97 (127.21.175.65:43767), 93aee57ce353431193229e1ca656f72a (127.21.175.66:45585)
I20260810 03:39:58.156201 23948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:58.152335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:43767 (local address 127.21.175.67:41283)
0810 03:39:58.152705 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:58.152722 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:58.152834 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:58.153138 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:58.153141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:58.153158 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:39:58.153284 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:58.153290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.154107 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.154111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:58.155043 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:58.155053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.155726 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.155730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:58.155827 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:58.155856 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 03:39:58.155950 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 03:39:58.156031 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":5,"thread_start_us":81,"threads_started":1}
I20260810 03:39:58.156188 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:58.152495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:41283 (local address 127.21.175.65:43767)
0810 03:39:58.152765 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:58.152772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:58.152789 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:39:58.152975 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:58.152978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:58.153022 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:58.153117 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:39:58.153403 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.153981 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.155178 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.155592 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.155729 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:58.155789 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:58.155967 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 03:39:58.156047 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":98,"threads_started":1}
I20260810 03:39:58.156381 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:58.152354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:45585 (local address 127.21.175.67:49759)
0810 03:39:58.152741 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:58.152757 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:58.152874 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:58.153186 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:58.153189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:58.153207 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:39:58.153352 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:58.153358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.154194 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.154197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:58.155124 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:58.155134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.155863 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.155867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:58.155990 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:58.156016 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:39:58.156123 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 03:39:58.156203 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":82,"threads_started":1}
I20260810 03:39:58.156602 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:58.152475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:49759 (local address 127.21.175.66:45585)
0810 03:39:58.152744 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:58.152752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:58.152819 (+ 67us) server_negotiation.cc:407] Connection header received
0810 03:39:58.153005 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:58.153009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:58.153068 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:58.153172 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:39:58.153472 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.154060 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.155268 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.155728 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.155872 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:58.155938 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:58.156144 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 03:39:58.156226 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":95,"threads_started":1}
I20260810 03:39:58.156713 23536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ea80a217b4b4ae6972108559ed3caa5" candidate_uuid: "09bcadd1a0484f94adb825885ee53688" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e42b8e37147c4a63bc393030b20d2e97" is_pre_election: true
I20260810 03:39:58.156896 23536 raft_consensus.cc:2454] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09bcadd1a0484f94adb825885ee53688 in term 0.
I20260810 03:39:58.157123 23679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ea80a217b4b4ae6972108559ed3caa5" candidate_uuid: "09bcadd1a0484f94adb825885ee53688" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93aee57ce353431193229e1ca656f72a" is_pre_election: true
I20260810 03:39:58.157270 23759 leader_election.cc:302] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09bcadd1a0484f94adb825885ee53688, e42b8e37147c4a63bc393030b20d2e97; no voters:
I20260810 03:39:58.157380 23679 raft_consensus.cc:2454] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09bcadd1a0484f94adb825885ee53688 in term 0.
I20260810 03:39:58.157680 23915 raft_consensus.cc:2781] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:39:58.157768 23915 raft_consensus.cc:482] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:39:58.157811 23915 raft_consensus.cc:3037] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:39:58.158836 23915 raft_consensus.cc:504] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:58.159188 23915 leader_election.cc:288] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [CANDIDATE]: Term 1 election: Requested vote from peers e42b8e37147c4a63bc393030b20d2e97 (127.21.175.65:43767), 93aee57ce353431193229e1ca656f72a (127.21.175.66:45585)
I20260810 03:39:58.159523 23536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ea80a217b4b4ae6972108559ed3caa5" candidate_uuid: "09bcadd1a0484f94adb825885ee53688" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e42b8e37147c4a63bc393030b20d2e97"
I20260810 03:39:58.159535 23679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ea80a217b4b4ae6972108559ed3caa5" candidate_uuid: "09bcadd1a0484f94adb825885ee53688" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93aee57ce353431193229e1ca656f72a"
I20260810 03:39:58.159682 23536 raft_consensus.cc:3037] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:39:58.159693 23679 raft_consensus.cc:3037] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:39:58.160631 23536 raft_consensus.cc:2454] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09bcadd1a0484f94adb825885ee53688 in term 1.
I20260810 03:39:58.160650 23679 raft_consensus.cc:2454] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09bcadd1a0484f94adb825885ee53688 in term 1.
I20260810 03:39:58.160983 23761 leader_election.cc:302] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09bcadd1a0484f94adb825885ee53688, 93aee57ce353431193229e1ca656f72a; no voters:
I20260810 03:39:58.161187 23915 raft_consensus.cc:2781] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:39:58.161360 23915 raft_consensus.cc:686] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [term 1 LEADER]: Becoming Leader. State: Replica: 09bcadd1a0484f94adb825885ee53688, State: Running, Role: LEADER
I20260810 03:39:58.161669 23915 consensus_queue.cc:237] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } }
I20260810 03:39:58.163192 23952 tablet_replica.cc:439] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688: TxnStatusTablet state changed. Reason: New leader 09bcadd1a0484f94adb825885ee53688. Latest consensus state: current_term: 1 leader_uuid: "09bcadd1a0484f94adb825885ee53688" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } }
I20260810 03:39:58.163412 23952 tablet_replica.cc:442] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:39:58.163657 23955 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 03:39:58.163961 23955 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 03:39:58.164812 23373 catalog_manager.cc:5495] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09bcadd1a0484f94adb825885ee53688 (127.21.175.67). New cstate: current_term: 1 leader_uuid: "09bcadd1a0484f94adb825885ee53688" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } health_report { overall_health: HEALTHY } } }
I20260810 03:39:58.199175 23958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:58.195282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45944 (local address 127.21.175.67:40129)
0810 03:39:58.195624 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:58.195630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:58.195642 (+ 12us) server_negotiation.cc:407] Connection header received
0810 03:39:58.195709 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:58.195711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:58.195764 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:58.195887 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:39:58.196135 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.196676 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.197597 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.197834 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.197949 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:58.198293 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:39:58.198738 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:39:58.198866 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:58.198936 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 03:39:58.199000 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":127,"threads_started":1}
I20260810 03:39:58.199157 23944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:58.195122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:40129 (local address 127.0.0.1:45944)
0810 03:39:58.195284 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:58.195302 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:58.195415 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:58.195896 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:58.195899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:58.195907 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:39:58.196021 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:58.196027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.196792 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.196796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:58.197412 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:58.197421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.197953 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.197956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:58.198064 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:58.198113 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:39:58.198836 (+ 723us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:39:58.198855 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 03:39:58.198927 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 03:39:58.199030 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 03:39:58.202744 23536 raft_consensus.cc:1261] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97 [term 1 FOLLOWER]: Refusing update from remote peer 09bcadd1a0484f94adb825885ee53688: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:39:58.202895 23679 raft_consensus.cc:1261] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a [term 1 FOLLOWER]: Refusing update from remote peer 09bcadd1a0484f94adb825885ee53688: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:39:58.203416 23952 consensus_queue.cc:1035] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:39:58.203649 23915 consensus_queue.cc:1035] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:39:58.205129 23913 tablet_replica.cc:439] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97: TxnStatusTablet state changed. Reason: New leader 09bcadd1a0484f94adb825885ee53688. Latest consensus state: current_term: 1 leader_uuid: "09bcadd1a0484f94adb825885ee53688" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } }
I20260810 03:39:58.205255 23913 tablet_replica.cc:442] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:39:58.205526 23928 tablet_replica.cc:439] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a: TxnStatusTablet state changed. Reason: New leader 09bcadd1a0484f94adb825885ee53688. Latest consensus state: current_term: 1 leader_uuid: "09bcadd1a0484f94adb825885ee53688" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } }
I20260810 03:39:58.205642 23928 tablet_replica.cc:442] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:39:58.207685 23952 tablet_replica.cc:439] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09bcadd1a0484f94adb825885ee53688" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } }
I20260810 03:39:58.207805 23952 tablet_replica.cc:442] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:39:58.207932 23952 tablet_replica.cc:439] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09bcadd1a0484f94adb825885ee53688" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } }
I20260810 03:39:58.208029 23952 tablet_replica.cc:442] T 0ea80a217b4b4ae6972108559ed3caa5 P 09bcadd1a0484f94adb825885ee53688: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:39:58.208724 23928 tablet_replica.cc:439] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09bcadd1a0484f94adb825885ee53688" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } }
I20260810 03:39:58.208844 23928 tablet_replica.cc:442] T 0ea80a217b4b4ae6972108559ed3caa5 P 93aee57ce353431193229e1ca656f72a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:39:58.209606 23913 tablet_replica.cc:439] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09bcadd1a0484f94adb825885ee53688" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 } } peers { permanent_uuid: "93aee57ce353431193229e1ca656f72a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45585 } } peers { permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 } } }
I20260810 03:39:58.209738 23913 tablet_replica.cc:442] T 0ea80a217b4b4ae6972108559ed3caa5 P e42b8e37147c4a63bc393030b20d2e97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:39:58.209626 23960 mvcc.cc:205] Tried to move back new op lower bound from 7316820779832487936 to 7316820779673309184. Current Snapshot: MvccSnapshot[applied={T|T < 7316820779832487936}]
I20260810 03:39:58.211450 23962 mvcc.cc:205] Tried to move back new op lower bound from 7316820779832487936 to 7316820779673309184. Current Snapshot: MvccSnapshot[applied={T|T < 7316820779832487936 or (T in {7316820779832487936})}]
I20260810 03:39:58.232189 23958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:58.228583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:57733 (local address 127.21.175.67:40129)
0810 03:39:58.228739 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:58.228745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:58.228759 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:39:58.228819 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:58.228822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:58.228867 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:58.228960 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:39:58.229273 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.229963 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.230945 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.231208 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.231333 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:58.231682 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:39:58.231773 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:39:58.231865 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:58.231963 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 03:39:58.232021 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:39:58.232204 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:58.228162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:40129 (local address 127.21.175.66:57733)
0810 03:39:58.228511 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:58.228531 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:58.228651 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:58.228988 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:58.228992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:58.229004 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:39:58.229138 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:58.229144 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.230093 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.230098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:58.230801 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:58.230811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.231351 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.231355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:58.231460 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:58.231512 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:39:58.231869 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:39:58.231878 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 03:39:58.231952 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 03:39:58.232056 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":134,"threads_started":1}
I20260810 03:39:58.241564 23834 raft_consensus.cc:1261] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688 [term 1 FOLLOWER]: Refusing update from remote peer 93aee57ce353431193229e1ca656f72a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:39:58.242074 23928 consensus_queue.cc:1035] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "09bcadd1a0484f94adb825885ee53688" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:39:58.242218 23536 raft_consensus.cc:1261] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97 [term 1 FOLLOWER]: Refusing update from remote peer 93aee57ce353431193229e1ca656f72a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:39:58.242730 23928 consensus_queue.cc:1035] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e42b8e37147c4a63bc393030b20d2e97" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:39:58.270234 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:58.265405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:45585 (local address 127.21.175.67:54319)
0810 03:39:58.265829 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0810 03:39:58.265846 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:39:58.265967 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:39:58.266250 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:39:58.266253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:39:58.266281 (+ 28us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:39:58.266514 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:58.266523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.267552 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.267556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:58.268628 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:39:58.268641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.269161 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.269164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:39:58.269274 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:58.269345 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:39:58.269850 (+ 505us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:39:58.269859 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 03:39:58.269936 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 03:39:58.270032 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":194,"threads_started":1}
I20260810 03:39:58.270388 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:58.265516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:54319 (local address 127.21.175.66:45585)
0810 03:39:58.265665 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:58.265670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:58.265917 (+ 247us) server_negotiation.cc:407] Connection header received
0810 03:39:58.266061 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:58.266064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:58.266115 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:58.266231 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:39:58.266659 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.267423 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.268789 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.269016 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.269114 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:58.269660 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:39:58.269746 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:39:58.269815 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:58.270129 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 03:39:58.270238 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 03:39:58.298141 23372 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:39:58.298347 23372 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:39:58.300757 23372 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee922360100a485eacd6de8a07db609b: Sending DeleteTablet for 3 replicas of tablet c0b361d0328047dab36863ad3301c748
I20260810 03:39:58.301504 23657 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b361d0328047dab36863ad3301c748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:39:58 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:56982
I20260810 03:39:58.301506 23812 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b361d0328047dab36863ad3301c748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:39:58 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:45928
I20260810 03:39:58.301514 23515 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b361d0328047dab36863ad3301c748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:39:58 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:59526
I20260810 03:39:58.302132 23981 tablet_replica.cc:330] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97: stopping tablet replica
I20260810 03:39:58.302295 23981 raft_consensus.cc:2229] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:39:58.302452 23981 raft_consensus.cc:2258] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:39:58.302703 23982 tablet_replica.cc:330] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a: stopping tablet replica
I20260810 03:39:58.302840 23982 raft_consensus.cc:2229] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:39:58.303073 23982 raft_consensus.cc:2258] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:39:58.303351 23980 tablet_replica.cc:330] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688: stopping tablet replica
I20260810 03:39:58.303511 23980 raft_consensus.cc:2229] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:39:58.303550 23981 ts_tablet_manager.cc:1905] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:39:58.303661 23980 raft_consensus.cc:2258] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:39:58.304392 23982 ts_tablet_manager.cc:1905] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:39:58.307366 23981 ts_tablet_manager.cc:1918] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:39:58.307832 23982 ts_tablet_manager.cc:1918] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:39:58.307929 23982 log.cc:1196] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/c0b361d0328047dab36863ad3301c748
I20260810 03:39:58.308066 23981 log.cc:1196] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-0/wal/wals/c0b361d0328047dab36863ad3301c748
I20260810 03:39:58.308256 23982 ts_tablet_manager.cc:1939] T c0b361d0328047dab36863ad3301c748 P 93aee57ce353431193229e1ca656f72a: Deleting consensus metadata
I20260810 03:39:58.308547 23981 ts_tablet_manager.cc:1939] T c0b361d0328047dab36863ad3301c748 P e42b8e37147c4a63bc393030b20d2e97: Deleting consensus metadata
I20260810 03:39:58.308822 23980 ts_tablet_manager.cc:1905] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:39:58.309147 23359 catalog_manager.cc:4841] TS 93aee57ce353431193229e1ca656f72a (127.21.175.66:45585): tablet c0b361d0328047dab36863ad3301c748 (table test-table [id=af40c7bc80314cf3bbd71939dbead758]) successfully deleted
I20260810 03:39:58.310266 23358 catalog_manager.cc:4841] TS e42b8e37147c4a63bc393030b20d2e97 (127.21.175.65:43767): tablet c0b361d0328047dab36863ad3301c748 (table test-table [id=af40c7bc80314cf3bbd71939dbead758]) successfully deleted
Aug 10 03:39:58 dist-test-slave-ss4n krb5kdc[23327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333197, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.175.65@KRBTEST.COM
I20260810 03:39:58.312489 23980 ts_tablet_manager.cc:1918] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:39:58.312609 23980 log.cc:1196] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/c0b361d0328047dab36863ad3301c748
I20260810 03:39:58.312994 23980 ts_tablet_manager.cc:1939] T c0b361d0328047dab36863ad3301c748 P 09bcadd1a0484f94adb825885ee53688: Deleting consensus metadata
I20260810 03:39:58.313863 23359 catalog_manager.cc:4841] TS 09bcadd1a0484f94adb825885ee53688 (127.21.175.67:40129): tablet c0b361d0328047dab36863ad3301c748 (table test-table [id=af40c7bc80314cf3bbd71939dbead758]) successfully deleted
I20260810 03:39:58.316300 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:58.303472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.21.175.65:43767)
0810 03:39:58.303588 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:58.303593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:58.304115 (+ 522us) server_negotiation.cc:407] Connection header received
0810 03:39:58.304302 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:58.304305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:58.304353 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:58.304405 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:58.305408 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.306190 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.307656 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.307994 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.308084 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:58.311896 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:58.311908 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:58.311925 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:58.311956 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:58.314229 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:58.314732 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:58.314738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:58.314740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:58.314909 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:58.315219 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:58.315224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:58.315225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:58.315488 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:58.315704 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:58.315882 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 03:39:58.316021 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 03:39:58.317051 23561 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59534
I20260810 03:39:58.329227 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:58.319142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42872 (local address 127.21.175.126:38579)
0810 03:39:58.319318 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:58.319323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:58.319610 (+ 287us) server_negotiation.cc:407] Connection header received
0810 03:39:58.319734 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:58.319738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:58.319795 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:58.319893 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:58.320783 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.321622 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.322695 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:58.323037 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:58.323172 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:58.324956 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:58.324961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:58.324964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:58.324999 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:58.327014 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:58.327540 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:58.327544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:58.327546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:58.327659 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:58.328076 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:58.328080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:58.328082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:58.328405 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:58.328574 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:58.328888 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 03:39:58.329029 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 03:39:58.329726 23372 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42872
I20260810 03:39:58.331069 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 23448
I20260810 03:39:58.340276 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 23592
I20260810 03:39:58.352633 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 23739
I20260810 03:39:58.362203 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 23340
2026-08-10T03:39:58Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3155 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:39:58 dist-test-slave-ss4n krb5kdc[23989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:39:58 dist-test-slave-ss4n krb5kdc[23989](info): set up 2 sockets
Aug 10 03:39:58 dist-test-slave-ss4n krb5kdc[23989](info): commencing operation
krb5kdc: starting...
W20260810 03:39:59.511919 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:39:59 dist-test-slave-ss4n krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:39:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:39:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:39:59.654474 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:42539
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:42539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:39:59.777599 24002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:39:59.777938 24002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:39:59.778020 24002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:39:59.782171 24002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:39:59.782277 24002 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 03:39:59.782312 24002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:39:59.782356 24002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:39:59.782392 24002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:39:59.782433 24002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:39:59.787999 24002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:42539
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:42539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:39:59.789242 24002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:39:59.790372 24002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:39:59.800267 24010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:59.800493 24002 server_base.cc:1034] running on GCE node
W20260810 03:39:59.800326 24007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:39:59.800267 24008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:39:59.801393 24002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:39:59.802807 24002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:39:59.804042 24002 hybrid_clock.cc:648] HybridClock initialized: now 1786333199804023 us; error 32 us; skew 500 ppm
Aug 10 03:39:59 dist-test-slave-ss4n krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333199, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:39:59.807560 24002 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:39:59.808537 24002 webserver.cc:460] Webserver started at http://127.21.175.126:36489/ using document root <none> and password file <none>
I20260810 03:39:59.809108 24002 fs_manager.cc:359] Metadata directory not provided
I20260810 03:39:59.809173 24002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:39:59.809360 24002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:39:59.811602 24002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "6ceef1a53d4249de9dbee1a00b7c91d0"
format_stamp: "Formatted at 2026-08-10 03:39:59 on dist-test-slave-ss4n"
server_key: "8392d3d87aa9d716a7021d355dcc2c85"
server_key_iv: "41fb3585dc15b68a3d26db502a904744"
server_key_version: "encryptionkey@0"
I20260810 03:39:59.812224 24002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "6ceef1a53d4249de9dbee1a00b7c91d0"
format_stamp: "Formatted at 2026-08-10 03:39:59 on dist-test-slave-ss4n"
server_key: "8392d3d87aa9d716a7021d355dcc2c85"
server_key_iv: "41fb3585dc15b68a3d26db502a904744"
server_key_version: "encryptionkey@0"
I20260810 03:39:59.816268 24002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:39:59.819140 24017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:39:59.820600 24002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:39:59.820765 24002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "6ceef1a53d4249de9dbee1a00b7c91d0"
format_stamp: "Formatted at 2026-08-10 03:39:59 on dist-test-slave-ss4n"
server_key: "8392d3d87aa9d716a7021d355dcc2c85"
server_key_iv: "41fb3585dc15b68a3d26db502a904744"
server_key_version: "encryptionkey@0"
I20260810 03:39:59.820894 24002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:39:59.844736 24002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:39:59.848670 24002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:39:59.848996 24002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:39:59.859417 24002 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:42539
I20260810 03:39:59.859431 24083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:42539 every 8 connection(s)
I20260810 03:39:59.860630 24002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:39:59.863265 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 24002
I20260810 03:39:59.863353 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:39:59.863682 22205 external_mini_cluster.cc:1428] Setting key a9b8f9f25083fd3c8d28371f77e606af
I20260810 03:39:59.863914 24084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:39:59.871208 24084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0: Bootstrap starting.
I20260810 03:39:59.874338 24084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0: Neither blocks nor log segments found. Creating new log.
Aug 10 03:39:59 dist-test-slave-ss4n krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:39:59.875231 24084 log.cc:824] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0: Log is configured to *not* fsync() on all Append() calls
I20260810 03:39:59.877662 24084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0: No bootstrap required, opened a new log
I20260810 03:39:59.880918 24084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ceef1a53d4249de9dbee1a00b7c91d0" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42539 } }
I20260810 03:39:59.881119 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:39:59.865295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.21.175.126:42539)
0810 03:39:59.865804 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0810 03:39:59.865821 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:39:59.865887 (+ 66us) server_negotiation.cc:407] Connection header received
0810 03:39:59.866951 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:39:59.866976 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:39:59.867364 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:39:59.867802 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:39:59.868778 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:59.870021 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:59.871270 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:39:59.871909 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:39:59.872048 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:39:59.875310 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:39:59.875320 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:39:59.875341 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:39:59.875401 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:39:59.878058 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:59.878561 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:59.878567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:59.878573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:59.878711 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:39:59.879039 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:39:59.879043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:39:59.879044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:39:59.879475 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:39:59.879709 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 03:39:59.880025 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 03:39:59.880337 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":188,"threads_started":1}
I20260810 03:39:59.881196 24084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:39:59.881278 24084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ceef1a53d4249de9dbee1a00b7c91d0, State: Initialized, Role: FOLLOWER
I20260810 03:39:59.881858 24084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ceef1a53d4249de9dbee1a00b7c91d0" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42539 } }
I20260810 03:39:59.882025 24084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:39:59.882088 24084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:39:59.882215 24084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:39:59.883198 24084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ceef1a53d4249de9dbee1a00b7c91d0" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42539 } }
I20260810 03:39:59.883606 24084 leader_election.cc:302] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ceef1a53d4249de9dbee1a00b7c91d0; no voters:
I20260810 03:39:59.883977 24084 leader_election.cc:288] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:39:59.884097 24089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:39:59.884361 24089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [term 1 LEADER]: Becoming Leader. State: Replica: 6ceef1a53d4249de9dbee1a00b7c91d0, State: Running, Role: LEADER
I20260810 03:39:59.884724 24089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ceef1a53d4249de9dbee1a00b7c91d0" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42539 } }
I20260810 03:39:59.885280 24084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:39:59.886461 24092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ceef1a53d4249de9dbee1a00b7c91d0. Latest consensus state: current_term: 1 leader_uuid: "6ceef1a53d4249de9dbee1a00b7c91d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ceef1a53d4249de9dbee1a00b7c91d0" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42539 } } }
I20260810 03:39:59.886613 24092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [sys.catalog]: This master's current role is: LEADER
I20260810 03:39:59.886801 24090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ceef1a53d4249de9dbee1a00b7c91d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ceef1a53d4249de9dbee1a00b7c91d0" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42539 } } }
I20260810 03:39:59.886973 24090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0 [sys.catalog]: This master's current role is: LEADER
W20260810 03:39:59.890561 24105 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:39:59.890673 24105 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:39:59.890749 24100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:39:59.891515 24100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:39:59.897872 24100 catalog_manager.cc:1347] Generated new cluster ID: 1035ccd047954750b29ac77e3598488e
I20260810 03:39:59.897986 24100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:39:59.904783 24100 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:39:59.905902 24100 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:39:59.913640 24100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0: Generated new TSK 0
I20260810 03:39:59.914597 24100 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:39:59.948805 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:42539
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:00.072088 24112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:00.072448 24112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:00.072540 24112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:00.077013 24112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:00.077118 24112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:00.077194 24112 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 03:40:00.077240 24112 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:40:00.077315 24112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:40:00.083065 24112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.175.126:42539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:00.084259 24112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:00.085373 24112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:00.096158 24118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:00.096218 24119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:00.096158 24121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:00.096415 24112 server_base.cc:1034] running on GCE node
I20260810 03:40:00.097059 24112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:00.097816 24112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:00.099007 24112 hybrid_clock.cc:648] HybridClock initialized: now 1786333200098977 us; error 44 us; skew 500 ppm
Aug 10 03:40:00 dist-test-slave-ss4n krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333200, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:00.102346 24112 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:40:00.103336 24112 webserver.cc:460] Webserver started at http://127.21.175.65:34649/ using document root <none> and password file <none>
I20260810 03:40:00.103958 24112 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:00.104056 24112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:00.104308 24112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:00.106508 24112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c"
format_stamp: "Formatted at 2026-08-10 03:40:00 on dist-test-slave-ss4n"
server_key: "8fc34ef27b8af518a0bf7e948784545f"
server_key_iv: "1f4080a32cb4b2c7b89b78e5e7ccda29"
server_key_version: "encryptionkey@0"
I20260810 03:40:00.107124 24112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c"
format_stamp: "Formatted at 2026-08-10 03:40:00 on dist-test-slave-ss4n"
server_key: "8fc34ef27b8af518a0bf7e948784545f"
server_key_iv: "1f4080a32cb4b2c7b89b78e5e7ccda29"
server_key_version: "encryptionkey@0"
I20260810 03:40:00.111090 24112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:40:00.113780 24128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:00.115020 24112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:40:00.115171 24112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c"
format_stamp: "Formatted at 2026-08-10 03:40:00 on dist-test-slave-ss4n"
server_key: "8fc34ef27b8af518a0bf7e948784545f"
server_key_iv: "1f4080a32cb4b2c7b89b78e5e7ccda29"
server_key_version: "encryptionkey@0"
I20260810 03:40:00.115304 24112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:00.150243 24112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:00.153556 24112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:00.153798 24112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:00.155349 24112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:00.155431 24112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:00.155504 24112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:00.155550 24112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:40:00 dist-test-slave-ss4n krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333200, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:00.166983 24112 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:40223
I20260810 03:40:00.167186 24253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:40223 every 8 connection(s)
I20260810 03:40:00.167994 24112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:40:00.170770 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.156385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:52553 (local address 127.21.175.126:42539)
0810 03:40:00.156534 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:00.156538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:00.157850 (+ 1312us) server_negotiation.cc:407] Connection header received
0810 03:40:00.159104 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:00.159108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:00.159174 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:00.159289 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:00.161163 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.161852 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.163270 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.163513 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.163635 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.166456 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:00.166459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:00.166462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:00.166489 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:00.168270 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:00.169031 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:00.169035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:00.169036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:00.169122 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:00.169509 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:00.169513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:00.169515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:00.169754 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:00.169874 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:00.170407 (+ 533us) server_negotiation.cc:299] Negotiation successful
0810 03:40:00.170563 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 03:40:00.171573 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.157018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:42539 (local address 127.21.175.65:52553)
0810 03:40:00.157680 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:00.157742 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:00.158870 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:00.159546 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:00.159559 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:00.160114 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:00.160923 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.160949 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.161989 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.161992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.163116 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.163126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.163697 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.163701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.163874 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.164515 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:00.164532 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:00.166259 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:00.168391 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:00.168398 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:00.168413 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:00.168766 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:00.169226 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:00.169230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:00.169232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:00.169383 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:00.169872 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:00.169879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:00.170147 (+ 268us) client_negotiation.cc:769] Sending connection context
0810 03:40:00.170360 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 03:40:00.170728 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":149,"threads_started":1}
I20260810 03:40:00.172941 24254 heartbeater.cc:347] Connected to a master server at 127.21.175.126:42539
I20260810 03:40:00.173118 24254 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:00.173704 24254 heartbeater.cc:511] Master 127.21.175.126:42539 requested a full tablet report, sending...
I20260810 03:40:00.175428 24033 ts_manager.cc:194] Registered new tserver with Master: ba4e73928c5a48efa9d906fe8aa8ba4c (127.21.175.65:40223)
I20260810 03:40:00.176657 24033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:52553
I20260810 03:40:00.178227 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 24112
I20260810 03:40:00.178381 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:40:00.178665 22205 external_mini_cluster.cc:1428] Setting key a5e964d851a0df328a9554beadae7e75
I20260810 03:40:00.185300 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.175407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:42587 (local address 127.21.175.126:42539)
0810 03:40:00.175558 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:00.175562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:00.177364 (+ 1802us) server_negotiation.cc:407] Connection header received
0810 03:40:00.177532 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:00.177536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:00.177582 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:00.177671 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:00.178315 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.179057 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.180189 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.180502 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.180631 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.181844 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:00.181850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:00.181856 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:00.181890 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:00.183465 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:00.183990 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:00.183993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:00.183994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:00.184080 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:00.184407 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:00.184409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:00.184410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:00.184624 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:00.184705 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:00.185058 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 03:40:00.185174 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:40:00.185550 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.175270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:42539 (local address 127.21.175.65:42587)
0810 03:40:00.177257 (+ 1987us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:00.177270 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:00.177408 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:00.177690 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:00.177693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:00.177948 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:00.178168 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.178180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.179206 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.179215 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.180063 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.180077 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.180637 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.180640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.180714 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.181287 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:00.181298 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:00.181705 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:00.183583 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:00.183586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:00.183589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:00.183879 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:00.184187 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:00.184193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:00.184197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:00.184312 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:00.184724 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:00.184730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:00.184855 (+ 125us) client_negotiation.cc:769] Sending connection context
0810 03:40:00.184990 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 03:40:00.185192 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1911,"mutex_wait_us":14,"thread_start_us":131,"threads_started":1}
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:40:00.210816 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:42539
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:00.330188 24263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:00.330516 24263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:00.330569 24263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:00.334836 24263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:00.334920 24263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:00.334964 24263 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 03:40:00.334980 24263 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:40:00.335012 24263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:40:00.340720 24263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.175.126:42539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:00.341944 24263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:00.343034 24263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:00.353669 24269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:00.353735 24272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:00.353812 24263 server_base.cc:1034] running on GCE node
W20260810 03:40:00.353669 24270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:00.354414 24263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:00.355116 24263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:00.356312 24263 hybrid_clock.cc:648] HybridClock initialized: now 1786333200356291 us; error 35 us; skew 500 ppm
Aug 10 03:40:00 dist-test-slave-ss4n krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333200, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:00.359452 24263 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:40:00.360522 24263 webserver.cc:460] Webserver started at http://127.21.175.66:45671/ using document root <none> and password file <none>
I20260810 03:40:00.361162 24263 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:00.361266 24263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:00.361593 24263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:00.363852 24263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d"
format_stamp: "Formatted at 2026-08-10 03:40:00 on dist-test-slave-ss4n"
server_key: "2c45437794ecf598f57e202e41cde8b3"
server_key_iv: "f8c8407687e4a4439e92d182fb2e70b0"
server_key_version: "encryptionkey@0"
I20260810 03:40:00.364501 24263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d"
format_stamp: "Formatted at 2026-08-10 03:40:00 on dist-test-slave-ss4n"
server_key: "2c45437794ecf598f57e202e41cde8b3"
server_key_iv: "f8c8407687e4a4439e92d182fb2e70b0"
server_key_version: "encryptionkey@0"
I20260810 03:40:00.368858 24263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 03:40:00.371906 24280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:00.373422 24263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:00.373620 24263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d"
format_stamp: "Formatted at 2026-08-10 03:40:00 on dist-test-slave-ss4n"
server_key: "2c45437794ecf598f57e202e41cde8b3"
server_key_iv: "f8c8407687e4a4439e92d182fb2e70b0"
server_key_version: "encryptionkey@0"
I20260810 03:40:00.373762 24263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:00.407702 24263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:00.411376 24263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:00.411638 24263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:00.413257 24263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:00.413340 24263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:00.413420 24263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:00.413465 24263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:00.424598 24263 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:34281
I20260810 03:40:00.424629 24401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:34281 every 8 connection(s)
Aug 10 03:40:00 dist-test-slave-ss4n krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333200, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:00.425770 24263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:40:00.429234 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 24263
I20260810 03:40:00.429330 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:40:00.429719 22205 external_mini_cluster.cc:1428] Setting key 066f695dbec6dfb2df540a046be7c299
I20260810 03:40:00.429960 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.414403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:46959 (local address 127.21.175.126:42539)
0810 03:40:00.414564 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:00.414570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:00.415450 (+ 880us) server_negotiation.cc:407] Connection header received
0810 03:40:00.416751 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:00.416757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:00.416828 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:00.416942 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:00.418719 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.419624 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.421414 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.421835 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.421966 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.425757 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:00.425761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:00.425765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:00.425799 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:00.427771 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:00.428299 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:00.428302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:00.428303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:00.428383 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:00.428706 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:00.428708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:00.428709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:00.428899 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:00.429024 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:00.429647 (+ 623us) server_negotiation.cc:299] Negotiation successful
0810 03:40:00.429798 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:40:00.430944 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.414814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:42539 (local address 127.21.175.66:46959)
0810 03:40:00.415297 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:00.415349 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:00.416469 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:00.417164 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:00.417171 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:00.417748 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:00.418503 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.418524 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.419772 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.419775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.421282 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.421292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.422124 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.422132 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.422410 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.423240 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:00.423289 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:00.425536 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:00.427894 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:00.427899 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:00.427913 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:00.428204 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:00.428468 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:00.428470 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:00.428471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:00.428624 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:00.429016 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:00.429022 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:00.429279 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 03:40:00.429627 (+ 348us) client_negotiation.cc:240] Negotiation successful
0810 03:40:00.430032 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":127,"threads_started":1}
I20260810 03:40:00.432523 24402 heartbeater.cc:347] Connected to a master server at 127.21.175.126:42539
I20260810 03:40:00.432688 24402 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:00.433338 24402 heartbeater.cc:511] Master 127.21.175.126:42539 requested a full tablet report, sending...
I20260810 03:40:00.434623 24033 ts_manager.cc:194] Registered new tserver with Master: 6b8d906fb1cb4ecdb5f846746c90aa2d (127.21.175.66:34281)
I20260810 03:40:00.435491 24033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:46959
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
I20260810 03:40:00.443815 24410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.435163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:42539 (local address 127.21.175.66:33097)
0810 03:40:00.435446 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:00.435459 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:00.435590 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:00.435850 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:00.435856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:00.436104 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:00.436272 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.436286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.437543 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.437550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.438646 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.438661 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.439161 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.439167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.439283 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.439886 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:00.439907 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:00.440397 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:00.442241 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:00.442244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:00.442246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:00.442469 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:00.442808 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:00.442814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:00.442819 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:00.442922 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:00.443349 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:00.443351 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:00.443434 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 03:40:00.443519 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 03:40:00.443640 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":119,"threads_started":1}
I20260810 03:40:00.443907 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.435061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:33097 (local address 127.21.175.126:42539)
0810 03:40:00.435186 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:00.435193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:00.435535 (+ 342us) server_negotiation.cc:407] Connection header received
0810 03:40:00.435687 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:00.435690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:00.435735 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:00.435822 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:00.436435 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.437333 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.438795 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.439018 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.439125 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.440548 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:00.440551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:00.440553 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:00.440579 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:00.442114 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:00.442609 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:00.442612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:00.442613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:00.442701 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:00.443024 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:00.443026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:00.443027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:00.443260 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:00.443341 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:00.443603 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 03:40:00.443711 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:40:00.463239 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:42539
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:00.585497 24412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:00.585853 24412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:00.585966 24412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:00.590420 24412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:00.590526 24412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:00.590605 24412 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 03:40:00.590651 24412 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:40:00.590723 24412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:40:00.596237 24412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.175.126:42539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:00.597601 24412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:00.598714 24412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:00.609400 24418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:00.609400 24417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:00.609536 24420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:00.609764 24412 server_base.cc:1034] running on GCE node
I20260810 03:40:00.610296 24412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:00.611001 24412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:00.612190 24412 hybrid_clock.cc:648] HybridClock initialized: now 1786333200612152 us; error 50 us; skew 500 ppm
Aug 10 03:40:00 dist-test-slave-ss4n krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333200, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:00.615206 24412 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:40:00.616151 24412 webserver.cc:460] Webserver started at http://127.21.175.67:46263/ using document root <none> and password file <none>
I20260810 03:40:00.616711 24412 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:00.616772 24412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:00.616943 24412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:00.619182 24412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604"
format_stamp: "Formatted at 2026-08-10 03:40:00 on dist-test-slave-ss4n"
server_key: "77b9de653e72a58fba1aabd23602a1e8"
server_key_iv: "4a3b369ed0cdc6ecaefd84536b0c580f"
server_key_version: "encryptionkey@0"
I20260810 03:40:00.619804 24412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604"
format_stamp: "Formatted at 2026-08-10 03:40:00 on dist-test-slave-ss4n"
server_key: "77b9de653e72a58fba1aabd23602a1e8"
server_key_iv: "4a3b369ed0cdc6ecaefd84536b0c580f"
server_key_version: "encryptionkey@0"
I20260810 03:40:00.624050 24412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:40:00.626936 24427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:00.628280 24412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:40:00.628438 24412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604"
format_stamp: "Formatted at 2026-08-10 03:40:00 on dist-test-slave-ss4n"
server_key: "77b9de653e72a58fba1aabd23602a1e8"
server_key_iv: "4a3b369ed0cdc6ecaefd84536b0c580f"
server_key_version: "encryptionkey@0"
I20260810 03:40:00.628578 24412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:00.647241 24412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:00.650578 24412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:00.650813 24412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:00.652468 24412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:00.652554 24412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:00.652626 24412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:00.652668 24412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:40:00 dist-test-slave-ss4n krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333200, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:00.664716 24412 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:40151
I20260810 03:40:00.664746 24546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:40151 every 8 connection(s)
I20260810 03:40:00.665859 24412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:40:00.668509 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.653370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:59963 (local address 127.21.175.126:42539)
0810 03:40:00.653602 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:00.653606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:00.655344 (+ 1738us) server_negotiation.cc:407] Connection header received
0810 03:40:00.656696 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:00.656699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:00.656766 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:00.656876 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:00.658244 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.658874 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.660286 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.660525 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.660647 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.664162 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:00.664170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:00.664175 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:00.664212 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:00.666070 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:00.666759 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:00.666763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:00.666764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:00.666852 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:00.667256 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:00.667260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:00.667261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:00.667547 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:00.667660 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:00.668207 (+ 547us) server_negotiation.cc:299] Negotiation successful
0810 03:40:00.668336 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:40:00.669366 24436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.654559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:42539 (local address 127.21.175.67:59963)
0810 03:40:00.655139 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:00.655187 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:00.656447 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:00.657086 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:00.657096 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:00.657523 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:00.658069 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.658085 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.659049 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.659054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.660148 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.660159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.660702 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.660706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.660886 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.661681 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:00.661709 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:00.663894 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:00.666199 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:00.666206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:00.666222 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:00.666625 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:00.666956 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:00.666959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:00.666961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:00.667105 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:00.667670 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:00.667675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:00.667936 (+ 261us) client_negotiation.cc:769] Sending connection context
0810 03:40:00.668102 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 03:40:00.668384 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":153,"threads_started":1}
I20260810 03:40:00.670825 24547 heartbeater.cc:347] Connected to a master server at 127.21.175.126:42539
I20260810 03:40:00.671017 24547 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:00.671666 24547 heartbeater.cc:511] Master 127.21.175.126:42539 requested a full tablet report, sending...
I20260810 03:40:00.672106 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 24412
I20260810 03:40:00.672230 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:40:00.672572 22205 external_mini_cluster.cc:1428] Setting key 5d93f44f14588fa5903081f81c288bc2
I20260810 03:40:00.673000 24034 ts_manager.cc:194] Registered new tserver with Master: 6cf0d4b649f94dad9c7eb89fb2d2f604 (127.21.175.67:40151)
I20260810 03:40:00.673900 24034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:59963
I20260810 03:40:00.675014 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:40:00.683169 24554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.674025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:42539 (local address 127.21.175.67:40393)
0810 03:40:00.674329 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:00.674347 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:00.674451 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:00.675059 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:00.675063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:00.675322 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:00.675451 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.675461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.676588 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.676591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.677817 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.677835 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.678313 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.678319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.678448 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.679071 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:00.679093 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:00.679626 (+ 533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:00.681445 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:00.681452 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:00.681454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:00.681807 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:00.682119 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:00.682121 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:00.682123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:00.682223 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:00.682698 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:00.682700 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:00.682785 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 03:40:00.682872 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 03:40:00.683006 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":122,"threads_started":1}
I20260810 03:40:00.683621 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.674215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:40393 (local address 127.21.175.126:42539)
0810 03:40:00.674770 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:00.674774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:00.674787 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:40:00.674825 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:00.674827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:00.674874 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:00.674957 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:00.675605 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.676421 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.677955 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.678185 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.678266 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.679783 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:00.679786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:00.679788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:00.679811 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:00.681325 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:00.681915 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:00.681919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:00.681921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:00.682016 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:00.682320 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:00.682326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:00.682331 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:00.682601 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:00.682710 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:00.682957 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 03:40:00.683099 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467}
Aug 10 03:40:00 dist-test-slave-ss4n krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:40:00 dist-test-slave-ss4n krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:00.706593 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.696951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.21.175.126:42539)
0810 03:40:00.697107 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:00.697110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:00.697122 (+ 12us) server_negotiation.cc:407] Connection header received
0810 03:40:00.697164 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:00.697167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:00.697209 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:00.697277 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:00.698150 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.698767 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.699654 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.699894 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.699999 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.703157 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:00.703160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:00.703163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:00.703195 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:00.704706 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:00.705199 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:00.705208 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:00.705209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:00.705295 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:00.705641 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:00.705644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:00.705646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:00.705861 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:00.706009 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:00.706274 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 03:40:00.706405 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:40:00.709935 24033 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:40:00.713070 24033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:40:00.728865 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.722509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46788 (local address 127.21.175.67:40151)
0810 03:40:00.723792 (+ 1283us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:00.723803 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:00.723825 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:40:00.723896 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:00.723902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:00.724097 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:00.724320 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:00.724557 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.726005 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.727531 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.728094 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.728235 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.728420 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:00.728587 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 03:40:00.728689 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":158,"threads_started":1}
I20260810 03:40:00.728850 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.722191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:40151 (local address 127.0.0.1:46788)
0810 03:40:00.723111 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:00.723168 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:00.723315 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:00.724293 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:00.724296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:00.724316 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:00.724426 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.724436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.726136 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.726139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.727415 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.727423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.728240 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.728243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.728355 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.728395 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 03:40:00.728545 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 03:40:00.728688 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":147,"spinlock_wait_cycles":896,"thread_start_us":108,"threads_started":1}
I20260810 03:40:00.729028 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.722995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:40223 (local address 127.0.0.1:52594)
0810 03:40:00.723568 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:00.723594 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:00.723675 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:00.724468 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:00.724471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:00.724488 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:00.724585 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.724591 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.726006 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.726009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.727423 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.727434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.728664 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.728670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.728761 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.728780 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 03:40:00.728842 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 03:40:00.728909 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":123,"thread_start_us":116,"threads_started":1}
I20260810 03:40:00.729185 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.722677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39834 (local address 127.21.175.66:34281)
0810 03:40:00.723473 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:00.723507 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:00.723534 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:40:00.723633 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:00.723641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:00.723876 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:00.724038 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:00.724378 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.725415 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.727258 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.728124 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.728240 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.728423 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:00.728925 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 03:40:00.729020 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":99,"threads_started":1}
I20260810 03:40:00.729383 24566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.722252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:34281 (local address 127.0.0.1:39834)
0810 03:40:00.723166 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:00.723181 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:00.723313 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:00.724047 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:00.724053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:00.724079 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:00.724232 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.724243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.725575 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.725579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.727094 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.727105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.728494 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.728498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.728586 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.728614 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 03:40:00.728690 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 03:40:00.728768 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":101,"spinlock_wait_cycles":14848,"thread_start_us":93,"threads_started":1}
I20260810 03:40:00.729461 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.723452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.21.175.65:40223)
0810 03:40:00.723861 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:00.723866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:00.723884 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:40:00.723945 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:00.723948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:00.724172 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:00.724326 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:00.724716 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.725867 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.727577 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.728351 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.728460 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.728598 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:00.729208 (+ 610us) server_negotiation.cc:299] Negotiation successful
0810 03:40:00.729322 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":88,"threads_started":1}
I20260810 03:40:00.732699 24331 tablet_service.cc:1467] Processing CreateTablet for tablet 158b5101c79a4be890569338acb4837c (DEFAULT_TABLE table=test-table [id=3d28888a95bf405c86548cc94256d22c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:00.732682 24179 tablet_service.cc:1467] Processing CreateTablet for tablet 158b5101c79a4be890569338acb4837c (DEFAULT_TABLE table=test-table [id=3d28888a95bf405c86548cc94256d22c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:00.733949 24179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 158b5101c79a4be890569338acb4837c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:00.733949 24331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 158b5101c79a4be890569338acb4837c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:00.735296 24478 tablet_service.cc:1467] Processing CreateTablet for tablet 158b5101c79a4be890569338acb4837c (DEFAULT_TABLE table=test-table [id=3d28888a95bf405c86548cc94256d22c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:00.736310 24478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 158b5101c79a4be890569338acb4837c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:00.742439 24572 tablet_bootstrap.cc:492] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c: Bootstrap starting.
I20260810 03:40:00.744215 24574 tablet_bootstrap.cc:492] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604: Bootstrap starting.
I20260810 03:40:00.744484 24573 tablet_bootstrap.cc:492] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d: Bootstrap starting.
I20260810 03:40:00.745828 24572 tablet_bootstrap.cc:654] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:00.746243 24574 tablet_bootstrap.cc:654] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:00.746634 24573 tablet_bootstrap.cc:654] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:00.746812 24572 log.cc:824] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:00.747044 24574 log.cc:824] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:00.747447 24573 log.cc:824] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:00.748989 24572 tablet_bootstrap.cc:492] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c: No bootstrap required, opened a new log
I20260810 03:40:00.749246 24572 ts_tablet_manager.cc:1397] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 03:40:00.749599 24573 tablet_bootstrap.cc:492] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d: No bootstrap required, opened a new log
I20260810 03:40:00.749821 24573 ts_tablet_manager.cc:1397] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260810 03:40:00.751246 24574 tablet_bootstrap.cc:492] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604: No bootstrap required, opened a new log
I20260810 03:40:00.751490 24574 ts_tablet_manager.cc:1397] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 03:40:00.753188 24572 raft_consensus.cc:348] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } }
I20260810 03:40:00.753468 24572 raft_consensus.cc:374] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:00.753545 24572 raft_consensus.cc:729] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba4e73928c5a48efa9d906fe8aa8ba4c, State: Initialized, Role: FOLLOWER
I20260810 03:40:00.754022 24572 consensus_queue.cc:260] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } }
I20260810 03:40:00.754361 24573 raft_consensus.cc:348] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } }
I20260810 03:40:00.754690 24573 raft_consensus.cc:374] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:00.754751 24573 raft_consensus.cc:729] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8d906fb1cb4ecdb5f846746c90aa2d, State: Initialized, Role: FOLLOWER
I20260810 03:40:00.754761 24254 heartbeater.cc:503] Master 127.21.175.126:42539 was elected leader, sending a full tablet report...
I20260810 03:40:00.755235 24572 ts_tablet_manager.cc:1428] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260810 03:40:00.755244 24573 consensus_queue.cc:260] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } }
I20260810 03:40:00.756031 24402 heartbeater.cc:503] Master 127.21.175.126:42539 was elected leader, sending a full tablet report...
I20260810 03:40:00.755985 24574 raft_consensus.cc:348] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } }
I20260810 03:40:00.756258 24574 raft_consensus.cc:374] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:00.756312 24574 raft_consensus.cc:729] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf0d4b649f94dad9c7eb89fb2d2f604, State: Initialized, Role: FOLLOWER
I20260810 03:40:00.756434 24573 ts_tablet_manager.cc:1428] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260810 03:40:00.756757 24574 consensus_queue.cc:260] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } }
I20260810 03:40:00.757570 24547 heartbeater.cc:503] Master 127.21.175.126:42539 was elected leader, sending a full tablet report...
I20260810 03:40:00.757853 24574 ts_tablet_manager.cc:1428] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 03:40:00.775998 24580 raft_consensus.cc:482] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:00.776281 24580 raft_consensus.cc:504] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } }
I20260810 03:40:00.777947 24580 leader_election.cc:288] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cf0d4b649f94dad9c7eb89fb2d2f604 (127.21.175.67:40151), ba4e73928c5a48efa9d906fe8aa8ba4c (127.21.175.65:40223)
I20260810 03:40:00.781865 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.778381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:40531 (local address 127.21.175.67:40151)
0810 03:40:00.778520 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:00.778525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:00.778541 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:40:00.778622 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:00.778625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:00.778685 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:00.778775 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:00.779158 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.779779 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.780857 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.781263 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.781401 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.781461 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:00.781667 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 03:40:00.781744 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:40:00.781865 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.778211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:40151 (local address 127.21.175.66:40531)
0810 03:40:00.778338 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:00.778353 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:00.778462 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:00.778812 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:00.778818 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:00.778843 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:00.778984 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.778990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.779906 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.779911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.780717 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.780726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.781391 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.781395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.781533 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.781553 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 03:40:00.781648 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 03:40:00.781725 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 03:40:00.782612 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158b5101c79a4be890569338acb4837c" candidate_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" is_pre_election: true
I20260810 03:40:00.782946 24500 raft_consensus.cc:2454] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8d906fb1cb4ecdb5f846746c90aa2d in term 0.
I20260810 03:40:00.783143 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.778470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:40223 (local address 127.21.175.66:48767)
0810 03:40:00.778939 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:00.778956 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:00.779080 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:00.779346 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:00.779348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:00.779364 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:00.779909 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.779917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.780777 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.780784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.781676 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.781687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.782666 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.782670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.782815 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.782839 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:40:00.782917 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 03:40:00.782994 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":149,"threads_started":1}
I20260810 03:40:00.783149 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.778756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:48767 (local address 127.21.175.65:40223)
0810 03:40:00.778888 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:00.778892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:00.779069 (+ 177us) server_negotiation.cc:407] Connection header received
0810 03:40:00.779185 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:00.779189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:00.779243 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:00.779321 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:00.780067 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.780648 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.781816 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.782506 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.782666 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.782720 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:00.782957 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 03:40:00.783035 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:40:00.783571 24282 leader_election.cc:302] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8d906fb1cb4ecdb5f846746c90aa2d, 6cf0d4b649f94dad9c7eb89fb2d2f604; no voters:
I20260810 03:40:00.783919 24580 raft_consensus.cc:2781] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:40:00.784041 24580 raft_consensus.cc:482] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:40:00.784101 24580 raft_consensus.cc:3037] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:00.784374 24204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158b5101c79a4be890569338acb4837c" candidate_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" is_pre_election: true
I20260810 03:40:00.784612 24204 raft_consensus.cc:2454] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8d906fb1cb4ecdb5f846746c90aa2d in term 0.
I20260810 03:40:00.785423 24580 raft_consensus.cc:504] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } }
I20260810 03:40:00.785856 24580 leader_election.cc:288] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [CANDIDATE]: Term 1 election: Requested vote from peers 6cf0d4b649f94dad9c7eb89fb2d2f604 (127.21.175.67:40151), ba4e73928c5a48efa9d906fe8aa8ba4c (127.21.175.65:40223)
I20260810 03:40:00.786301 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158b5101c79a4be890569338acb4837c" candidate_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604"
I20260810 03:40:00.786306 24204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158b5101c79a4be890569338acb4837c" candidate_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c"
I20260810 03:40:00.786444 24204 raft_consensus.cc:3037] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:00.786443 24500 raft_consensus.cc:3037] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:00.787825 24500 raft_consensus.cc:2454] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8d906fb1cb4ecdb5f846746c90aa2d in term 1.
I20260810 03:40:00.787825 24204 raft_consensus.cc:2454] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8d906fb1cb4ecdb5f846746c90aa2d in term 1.
I20260810 03:40:00.788148 24282 leader_election.cc:302] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8d906fb1cb4ecdb5f846746c90aa2d, 6cf0d4b649f94dad9c7eb89fb2d2f604; no voters:
I20260810 03:40:00.788349 24580 raft_consensus.cc:2781] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:00.788600 24580 raft_consensus.cc:686] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 LEADER]: Becoming Leader. State: Replica: 6b8d906fb1cb4ecdb5f846746c90aa2d, State: Running, Role: LEADER
I20260810 03:40:00.788949 24580 consensus_queue.cc:237] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } }
I20260810 03:40:00.792410 24033 catalog_manager.cc:5495] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b8d906fb1cb4ecdb5f846746c90aa2d (127.21.175.66). New cstate: current_term: 1 leader_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } health_report { overall_health: UNKNOWN } } }
Aug 10 03:40:00 dist-test-slave-ss4n krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333199, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:00.840871 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.831420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.21.175.126:42539)
0810 03:40:00.831614 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:00.831618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:00.831746 (+ 128us) server_negotiation.cc:407] Connection header received
0810 03:40:00.831906 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:00.831909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:00.831958 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:00.832049 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:00.832864 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.833390 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.834302 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.834544 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.834662 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.837290 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:00.837294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:00.837299 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:00.837326 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:00.838973 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:00.839473 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:00.839479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:00.839481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:00.839563 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:00.839885 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:00.839890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:00.839892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:00.840082 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:00.840189 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:00.840501 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 03:40:00.840704 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":110}
I20260810 03:40:00.840873 24591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:00.831305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:42539 (local address 127.0.0.1:51604)
0810 03:40:00.831638 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:00.831652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:00.831806 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:00.832071 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:00.832074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:00.832583 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:00.832744 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.832752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.833531 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.833536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.834173 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:00.834183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:00.834680 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:00.834683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:00.834841 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:00.835429 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:00.835441 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:00.837123 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:00.839107 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:00.839118 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:00.839126 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:00.839367 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:00.839701 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:00.839704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:00.839706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:00.839794 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:00.840202 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:00.840211 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:00.840331 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 03:40:00.840451 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 03:40:00.840646 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":43,"thread_start_us":104,"threads_started":1}
I20260810 03:40:00.845213 24033 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:51604:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 03:40:00.846028 24033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:40:00.851222 24179 tablet_service.cc:1467] Processing CreateTablet for tablet e0fa892c724c4d52816036e2ded4158a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aee67af7b5bd4faa8e573f3900fd0a70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:40:00.851222 24331 tablet_service.cc:1467] Processing CreateTablet for tablet e0fa892c724c4d52816036e2ded4158a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aee67af7b5bd4faa8e573f3900fd0a70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:40:00.852039 24331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fa892c724c4d52816036e2ded4158a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:00.852038 24179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fa892c724c4d52816036e2ded4158a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:00.851675 24478 tablet_service.cc:1467] Processing CreateTablet for tablet e0fa892c724c4d52816036e2ded4158a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aee67af7b5bd4faa8e573f3900fd0a70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:40:00.852383 24478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fa892c724c4d52816036e2ded4158a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:00.855115 24574 tablet_bootstrap.cc:492] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604: Bootstrap starting.
I20260810 03:40:00.855120 24573 tablet_bootstrap.cc:492] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d: Bootstrap starting.
I20260810 03:40:00.855124 24572 tablet_bootstrap.cc:492] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c: Bootstrap starting.
I20260810 03:40:00.856096 24574 tablet_bootstrap.cc:654] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:00.856123 24573 tablet_bootstrap.cc:654] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:00.856127 24572 tablet_bootstrap.cc:654] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:00.857822 24572 tablet_bootstrap.cc:492] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c: No bootstrap required, opened a new log
I20260810 03:40:00.857944 24572 ts_tablet_manager.cc:1397] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260810 03:40:00.857981 24574 tablet_bootstrap.cc:492] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604: No bootstrap required, opened a new log
I20260810 03:40:00.858067 24574 ts_tablet_manager.cc:1397] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:00.858086 24573 tablet_bootstrap.cc:492] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d: No bootstrap required, opened a new log
I20260810 03:40:00.858172 24573 ts_tablet_manager.cc:1397] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260810 03:40:00.858495 24574 raft_consensus.cc:348] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:00.858489 24572 raft_consensus.cc:348] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:00.858650 24572 raft_consensus.cc:374] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:00.858650 24574 raft_consensus.cc:374] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:00.858696 24572 raft_consensus.cc:729] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba4e73928c5a48efa9d906fe8aa8ba4c, State: Initialized, Role: FOLLOWER
I20260810 03:40:00.858696 24574 raft_consensus.cc:729] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf0d4b649f94dad9c7eb89fb2d2f604, State: Initialized, Role: FOLLOWER
I20260810 03:40:00.858659 24573 raft_consensus.cc:348] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:00.858785 24573 raft_consensus.cc:374] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:00.858822 24573 raft_consensus.cc:729] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8d906fb1cb4ecdb5f846746c90aa2d, State: Initialized, Role: FOLLOWER
I20260810 03:40:00.858829 24572 consensus_queue.cc:260] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:00.858829 24574 consensus_queue.cc:260] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:00.858950 24573 consensus_queue.cc:260] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:00.859109 24574 ts_tablet_manager.cc:1428] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 03:40:00.859120 24572 ts_tablet_manager.cc:1428] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 03:40:00.859194 24573 ts_tablet_manager.cc:1428] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260810 03:40:00.859375 24579 tablet_replica.cc:439] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } }
I20260810 03:40:00.859509 24579 tablet_replica.cc:442] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:40:00.859437 24581 tablet_replica.cc:439] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } }
I20260810 03:40:00.859570 24581 tablet_replica.cc:442] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:40:00.859844 24580 tablet_replica.cc:439] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } }
I20260810 03:40:00.859956 24580 tablet_replica.cc:442] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 03:40:00.917011 24549 tablet.cc:2369] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:40:00.919407 24255 tablet.cc:2369] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:40:00.927881 24403 tablet.cc:2369] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:40:01.107470 24580 raft_consensus.cc:482] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:01.107661 24580 raft_consensus.cc:504] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:01.108117 24580 leader_election.cc:288] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba4e73928c5a48efa9d906fe8aa8ba4c (127.21.175.65:40223), 6cf0d4b649f94dad9c7eb89fb2d2f604 (127.21.175.67:40151)
I20260810 03:40:01.108706 24204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fa892c724c4d52816036e2ded4158a" candidate_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" is_pre_election: true
I20260810 03:40:01.108780 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fa892c724c4d52816036e2ded4158a" candidate_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" is_pre_election: true
I20260810 03:40:01.108881 24204 raft_consensus.cc:2454] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8d906fb1cb4ecdb5f846746c90aa2d in term 0.
I20260810 03:40:01.108911 24500 raft_consensus.cc:2454] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8d906fb1cb4ecdb5f846746c90aa2d in term 0.
I20260810 03:40:01.109239 24282 leader_election.cc:302] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8d906fb1cb4ecdb5f846746c90aa2d, ba4e73928c5a48efa9d906fe8aa8ba4c; no voters:
I20260810 03:40:01.109419 24580 raft_consensus.cc:2781] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:40:01.109513 24580 raft_consensus.cc:482] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:40:01.109565 24580 raft_consensus.cc:3037] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:01.110582 24580 raft_consensus.cc:504] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:01.110991 24580 leader_election.cc:288] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [CANDIDATE]: Term 1 election: Requested vote from peers ba4e73928c5a48efa9d906fe8aa8ba4c (127.21.175.65:40223), 6cf0d4b649f94dad9c7eb89fb2d2f604 (127.21.175.67:40151)
I20260810 03:40:01.111433 24204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fa892c724c4d52816036e2ded4158a" candidate_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c"
I20260810 03:40:01.111546 24204 raft_consensus.cc:3037] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:01.111495 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fa892c724c4d52816036e2ded4158a" candidate_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604"
I20260810 03:40:01.111694 24500 raft_consensus.cc:3037] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:01.112612 24204 raft_consensus.cc:2454] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8d906fb1cb4ecdb5f846746c90aa2d in term 1.
I20260810 03:40:01.112758 24500 raft_consensus.cc:2454] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8d906fb1cb4ecdb5f846746c90aa2d in term 1.
I20260810 03:40:01.112944 24282 leader_election.cc:302] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8d906fb1cb4ecdb5f846746c90aa2d, ba4e73928c5a48efa9d906fe8aa8ba4c; no voters:
I20260810 03:40:01.113123 24580 raft_consensus.cc:2781] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:01.113238 24580 raft_consensus.cc:686] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 LEADER]: Becoming Leader. State: Replica: 6b8d906fb1cb4ecdb5f846746c90aa2d, State: Running, Role: LEADER
I20260810 03:40:01.113394 24580 consensus_queue.cc:237] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:01.114135 24583 tablet_replica.cc:439] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d: TxnStatusTablet state changed. Reason: New leader 6b8d906fb1cb4ecdb5f846746c90aa2d. Latest consensus state: current_term: 1 leader_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } }
I20260810 03:40:01.114259 24583 tablet_replica.cc:442] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:40:01.114521 24596 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 03:40:01.114780 24596 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 03:40:01.115346 24032 catalog_manager.cc:5495] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b8d906fb1cb4ecdb5f846746c90aa2d (127.21.175.66). New cstate: current_term: 1 leader_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } health_report { overall_health: UNKNOWN } } }
I20260810 03:40:01.160996 24591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:01.156675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:34281 (local address 127.0.0.1:39850)
0810 03:40:01.156890 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:01.156913 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:01.157009 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:01.157370 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:01.157372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:01.157378 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:40:01.157473 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:01.157478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:01.158515 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:01.158518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:01.159142 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:01.159151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:01.159737 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:01.159740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:01.159818 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:01.159873 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:40:01.160725 (+ 852us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:40:01.160739 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 03:40:01.160791 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 03:40:01.160873 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 03:40:01.161022 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:01.156814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39850 (local address 127.21.175.66:34281)
0810 03:40:01.156963 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:01.156968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:01.156987 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:40:01.157171 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:01.157175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:01.157246 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:01.157353 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:40:01.157580 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:01.158371 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:01.159269 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:01.159619 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:01.159726 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:01.160026 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:40:01.160624 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:40:01.160744 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:01.160820 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 03:40:01.160886 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 03:40:01.166851 24204 raft_consensus.cc:1261] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Refusing update from remote peer 6b8d906fb1cb4ecdb5f846746c90aa2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:01.166858 24500 raft_consensus.cc:1261] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Refusing update from remote peer 6b8d906fb1cb4ecdb5f846746c90aa2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:01.167641 24583 consensus_queue.cc:1035] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:01.168133 24580 consensus_queue.cc:1035] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:01.173029 24579 tablet_replica.cc:439] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c: TxnStatusTablet state changed. Reason: New leader 6b8d906fb1cb4ecdb5f846746c90aa2d. Latest consensus state: current_term: 1 leader_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } }
I20260810 03:40:01.173192 24579 tablet_replica.cc:442] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:40:01.173542 24581 tablet_replica.cc:439] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604: TxnStatusTablet state changed. Reason: New leader 6b8d906fb1cb4ecdb5f846746c90aa2d. Latest consensus state: current_term: 1 leader_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } }
I20260810 03:40:01.173667 24581 tablet_replica.cc:442] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:40:01.178335 24583 tablet_replica.cc:439] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } }
I20260810 03:40:01.178463 24583 tablet_replica.cc:442] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:40:01.178403 24600 mvcc.cc:205] Tried to move back new op lower bound from 7316820791971504128 to 7316820791762063360. Current Snapshot: MvccSnapshot[applied={T|T < 7316820791971504128}]
I20260810 03:40:01.180187 24579 tablet_replica.cc:439] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } }
I20260810 03:40:01.180307 24579 tablet_replica.cc:442] T e0fa892c724c4d52816036e2ded4158a P ba4e73928c5a48efa9d906fe8aa8ba4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:40:01.181865 24602 mvcc.cc:205] Tried to move back new op lower bound from 7316820791971504128 to 7316820791762063360. Current Snapshot: MvccSnapshot[applied={T|T < 7316820791971504128}]
I20260810 03:40:01.183671 24580 tablet_replica.cc:439] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } }
I20260810 03:40:01.183789 24580 tablet_replica.cc:442] T e0fa892c724c4d52816036e2ded4158a P 6b8d906fb1cb4ecdb5f846746c90aa2d: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:40:01.185014 24581 tablet_replica.cc:439] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } } }
I20260810 03:40:01.185141 24581 tablet_replica.cc:442] T e0fa892c724c4d52816036e2ded4158a P 6cf0d4b649f94dad9c7eb89fb2d2f604: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:40:01.185663 24601 mvcc.cc:205] Tried to move back new op lower bound from 7316820791971504128 to 7316820791762063360. Current Snapshot: MvccSnapshot[applied={T|T < 7316820791971504128}]
I20260810 03:40:01.190433 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:01.186412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.21.175.66:34281)
0810 03:40:01.187098 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:01.187103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:01.187118 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:40:01.187165 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:01.187168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:01.187217 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:01.187300 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:40:01.187707 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:01.188416 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:01.189304 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:01.189617 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:01.189727 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:01.189998 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:40:01.190079 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:40:01.190158 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:01.190251 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 03:40:01.190301 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 03:40:01.206708 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:01.203464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:60867 (local address 127.21.175.66:34281)
0810 03:40:01.203566 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:01.203570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:01.203583 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:40:01.203721 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:01.203724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:01.203768 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:01.203847 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:40:01.204094 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:01.204588 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:01.205449 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:01.205718 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:01.205828 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:01.206158 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:40:01.206243 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:40:01.206287 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:01.206426 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 03:40:01.206545 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":47}
I20260810 03:40:01.206717 24613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:01.203136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:34281 (local address 127.21.175.66:60867)
0810 03:40:01.203439 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:01.203456 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:01.203563 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:01.203865 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:01.203868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:01.203875 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:40:01.203984 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:01.203991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:01.204704 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:01.204710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:01.205334 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:01.205342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:01.205827 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:01.205831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:01.205918 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:01.205971 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:40:01.206333 (+ 362us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:40:01.206337 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 03:40:01.206415 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 03:40:01.206569 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":46,"thread_start_us":115,"threads_started":1}
I20260810 03:40:01.214579 24204 raft_consensus.cc:1261] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Refusing update from remote peer 6b8d906fb1cb4ecdb5f846746c90aa2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:01.214613 24500 raft_consensus.cc:1261] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Refusing update from remote peer 6b8d906fb1cb4ecdb5f846746c90aa2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:01.215036 24583 consensus_queue.cc:1035] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:01.215181 24607 consensus_queue.cc:1035] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:01.265890 24033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:01.266110 24033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:01.268937 24033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0: Sending DeleteTablet for 3 replicas of tablet 158b5101c79a4be890569338acb4837c
I20260810 03:40:01.269797 24478 tablet_service.cc:1514] Processing DeleteTablet for tablet 158b5101c79a4be890569338acb4837c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:01 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:46788
I20260810 03:40:01.269816 24330 tablet_service.cc:1514] Processing DeleteTablet for tablet 158b5101c79a4be890569338acb4837c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:01 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:39834
I20260810 03:40:01.270327 24618 tablet_replica.cc:330] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604: stopping tablet replica
I20260810 03:40:01.270748 24179 tablet_service.cc:1514] Processing DeleteTablet for tablet 158b5101c79a4be890569338acb4837c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:01 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:52594
I20260810 03:40:01.270816 24618 raft_consensus.cc:2229] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:01.271193 24618 raft_consensus.cc:2258] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:01.271868 24620 tablet_replica.cc:330] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c: stopping tablet replica
I20260810 03:40:01.272238 24033 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:40:01.272780 24033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:40:01.272806 24619 tablet_replica.cc:330] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d: stopping tablet replica
I20260810 03:40:01.273977 24619 raft_consensus.cc:2229] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:40:01.274458 24619 raft_consensus.cc:2258] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:01.273608 24618 ts_tablet_manager.cc:1905] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:01.278256 24619 ts_tablet_manager.cc:1905] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:01.279809 24330 tablet_service.cc:1467] Processing CreateTablet for tablet 8b32133f6264493780a62ac750260f02 (DEFAULT_TABLE table=test-table [id=adaac25ec3a540ce9e27690f9558bbe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:01.280520 24618 ts_tablet_manager.cc:1918] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:40:01.280644 24618 log.cc:1196] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/158b5101c79a4be890569338acb4837c
I20260810 03:40:01.281004 24618 ts_tablet_manager.cc:1939] T 158b5101c79a4be890569338acb4837c P 6cf0d4b649f94dad9c7eb89fb2d2f604: Deleting consensus metadata
I20260810 03:40:01.281126 24179 tablet_service.cc:1467] Processing CreateTablet for tablet 8b32133f6264493780a62ac750260f02 (DEFAULT_TABLE table=test-table [id=adaac25ec3a540ce9e27690f9558bbe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:01.281338 24330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b32133f6264493780a62ac750260f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:01.281414 24179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b32133f6264493780a62ac750260f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:01.282251 24620 raft_consensus.cc:2229] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:01.282559 24620 raft_consensus.cc:2258] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:01.284153 24573 tablet_bootstrap.cc:492] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d: Bootstrap starting.
I20260810 03:40:01.284317 24572 tablet_bootstrap.cc:492] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c: Bootstrap starting.
I20260810 03:40:01.285192 24572 tablet_bootstrap.cc:654] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:01.285190 24573 tablet_bootstrap.cc:654] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:01.286234 24478 tablet_service.cc:1467] Processing CreateTablet for tablet 8b32133f6264493780a62ac750260f02 (DEFAULT_TABLE table=test-table [id=adaac25ec3a540ce9e27690f9558bbe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:01.286551 24478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b32133f6264493780a62ac750260f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:01.289134 24574 tablet_bootstrap.cc:492] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604: Bootstrap starting.
I20260810 03:40:01.289788 24619 ts_tablet_manager.cc:1918] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:40:01.289884 24619 log.cc:1196] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/158b5101c79a4be890569338acb4837c
I20260810 03:40:01.290143 24574 tablet_bootstrap.cc:654] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:01.290249 24619 ts_tablet_manager.cc:1939] T 158b5101c79a4be890569338acb4837c P 6b8d906fb1cb4ecdb5f846746c90aa2d: Deleting consensus metadata
I20260810 03:40:01.291510 24018 catalog_manager.cc:4841] TS 6b8d906fb1cb4ecdb5f846746c90aa2d (127.21.175.66:34281): tablet 158b5101c79a4be890569338acb4837c (table test-table [id=3d28888a95bf405c86548cc94256d22c]) successfully deleted
I20260810 03:40:01.292083 24019 catalog_manager.cc:4841] TS 6cf0d4b649f94dad9c7eb89fb2d2f604 (127.21.175.67:40151): tablet 158b5101c79a4be890569338acb4837c (table test-table [id=3d28888a95bf405c86548cc94256d22c]) successfully deleted
I20260810 03:40:01.292817 24573 tablet_bootstrap.cc:492] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d: No bootstrap required, opened a new log
I20260810 03:40:01.292918 24573 ts_tablet_manager.cc:1397] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20260810 03:40:01.293430 24573 raft_consensus.cc:348] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:01.293661 24573 raft_consensus.cc:374] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:01.293706 24573 raft_consensus.cc:729] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8d906fb1cb4ecdb5f846746c90aa2d, State: Initialized, Role: FOLLOWER
I20260810 03:40:01.293838 24573 consensus_queue.cc:260] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:01.294145 24573 ts_tablet_manager.cc:1428] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260810 03:40:01.294792 24572 tablet_bootstrap.cc:492] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c: No bootstrap required, opened a new log
I20260810 03:40:01.294920 24572 ts_tablet_manager.cc:1397] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260810 03:40:01.295439 24572 raft_consensus.cc:348] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:01.295531 24572 raft_consensus.cc:374] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:01.295552 24572 raft_consensus.cc:729] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba4e73928c5a48efa9d906fe8aa8ba4c, State: Initialized, Role: FOLLOWER
I20260810 03:40:01.295710 24572 consensus_queue.cc:260] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:01.296422 24572 ts_tablet_manager.cc:1428] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:40:01.300870 24620 ts_tablet_manager.cc:1905] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:01.303663 24620 ts_tablet_manager.cc:1918] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:40:01.303864 24620 log.cc:1196] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/wal/wals/158b5101c79a4be890569338acb4837c
I20260810 03:40:01.304380 24620 ts_tablet_manager.cc:1939] T 158b5101c79a4be890569338acb4837c P ba4e73928c5a48efa9d906fe8aa8ba4c: Deleting consensus metadata
I20260810 03:40:01.305037 24574 tablet_bootstrap.cc:492] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604: No bootstrap required, opened a new log
I20260810 03:40:01.305150 24574 ts_tablet_manager.cc:1397] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20260810 03:40:01.305696 24574 raft_consensus.cc:348] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:01.305842 24574 raft_consensus.cc:374] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:01.305891 24574 raft_consensus.cc:729] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf0d4b649f94dad9c7eb89fb2d2f604, State: Initialized, Role: FOLLOWER
I20260810 03:40:01.306056 24574 consensus_queue.cc:260] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:01.306157 24019 catalog_manager.cc:4841] TS ba4e73928c5a48efa9d906fe8aa8ba4c (127.21.175.65:40223): tablet 158b5101c79a4be890569338acb4837c (table test-table [id=3d28888a95bf405c86548cc94256d22c]) successfully deleted
I20260810 03:40:01.306366 24574 ts_tablet_manager.cc:1428] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260810 03:40:01.750247 24580 raft_consensus.cc:482] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:01.750411 24580 raft_consensus.cc:504] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:01.750962 24580 leader_election.cc:288] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba4e73928c5a48efa9d906fe8aa8ba4c (127.21.175.65:40223), 6cf0d4b649f94dad9c7eb89fb2d2f604 (127.21.175.67:40151)
I20260810 03:40:01.751629 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b32133f6264493780a62ac750260f02" candidate_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" is_pre_election: true
I20260810 03:40:01.751629 24204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b32133f6264493780a62ac750260f02" candidate_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" is_pre_election: true
I20260810 03:40:01.751791 24204 raft_consensus.cc:2454] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8d906fb1cb4ecdb5f846746c90aa2d in term 0.
I20260810 03:40:01.751791 24500 raft_consensus.cc:2454] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8d906fb1cb4ecdb5f846746c90aa2d in term 0.
I20260810 03:40:01.752216 24282 leader_election.cc:302] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8d906fb1cb4ecdb5f846746c90aa2d, ba4e73928c5a48efa9d906fe8aa8ba4c; no voters:
I20260810 03:40:01.752408 24580 raft_consensus.cc:2781] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:40:01.752471 24580 raft_consensus.cc:482] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:40:01.752512 24580 raft_consensus.cc:3037] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:01.753585 24580 raft_consensus.cc:504] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:01.753938 24580 leader_election.cc:288] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [CANDIDATE]: Term 1 election: Requested vote from peers ba4e73928c5a48efa9d906fe8aa8ba4c (127.21.175.65:40223), 6cf0d4b649f94dad9c7eb89fb2d2f604 (127.21.175.67:40151)
I20260810 03:40:01.754331 24204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b32133f6264493780a62ac750260f02" candidate_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c"
I20260810 03:40:01.754397 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b32133f6264493780a62ac750260f02" candidate_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604"
I20260810 03:40:01.754518 24204 raft_consensus.cc:3037] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:01.754539 24500 raft_consensus.cc:3037] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:01.755492 24204 raft_consensus.cc:2454] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8d906fb1cb4ecdb5f846746c90aa2d in term 1.
I20260810 03:40:01.755515 24500 raft_consensus.cc:2454] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8d906fb1cb4ecdb5f846746c90aa2d in term 1.
I20260810 03:40:01.755856 24282 leader_election.cc:302] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8d906fb1cb4ecdb5f846746c90aa2d, ba4e73928c5a48efa9d906fe8aa8ba4c; no voters:
I20260810 03:40:01.756031 24580 raft_consensus.cc:2781] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:01.756126 24580 raft_consensus.cc:686] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 LEADER]: Becoming Leader. State: Replica: 6b8d906fb1cb4ecdb5f846746c90aa2d, State: Running, Role: LEADER
I20260810 03:40:01.756280 24580 consensus_queue.cc:237] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:01.758141 24033 catalog_manager.cc:5495] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b8d906fb1cb4ecdb5f846746c90aa2d (127.21.175.66). New cstate: current_term: 1 leader_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } health_report { overall_health: UNKNOWN } } }
I20260810 03:40:01.879614 24500 raft_consensus.cc:1261] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Refusing update from remote peer 6b8d906fb1cb4ecdb5f846746c90aa2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:01.879690 24204 raft_consensus.cc:1261] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Refusing update from remote peer 6b8d906fb1cb4ecdb5f846746c90aa2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:01.880066 24629 consensus_queue.cc:1035] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:01.880239 24607 consensus_queue.cc:1035] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 03:40:01.897341 24625 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 03:40:03.460979 24033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:03.461148 24033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:03.464152 24033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0: Sending DeleteTablet for 3 replicas of tablet 8b32133f6264493780a62ac750260f02
I20260810 03:40:03.464920 24179 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b32133f6264493780a62ac750260f02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:03 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:52594
I20260810 03:40:03.464955 24330 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b32133f6264493780a62ac750260f02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:03 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:39834
I20260810 03:40:03.465139 24478 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b32133f6264493780a62ac750260f02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:03 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:46788
I20260810 03:40:03.465504 24661 tablet_replica.cc:330] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c: stopping tablet replica
I20260810 03:40:03.465415 24662 tablet_replica.cc:330] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d: stopping tablet replica
I20260810 03:40:03.465624 24663 tablet_replica.cc:330] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604: stopping tablet replica
I20260810 03:40:03.465730 24661 raft_consensus.cc:2229] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:03.465782 24663 raft_consensus.cc:2229] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:03.465888 24662 raft_consensus.cc:2229] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:40:03.465919 24661 raft_consensus.cc:2258] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:03.465943 24663 raft_consensus.cc:2258] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:03.466181 24662 raft_consensus.cc:2258] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:03.467022 24662 ts_tablet_manager.cc:1905] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:03.469106 24661 ts_tablet_manager.cc:1905] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:03.469779 24663 ts_tablet_manager.cc:1905] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:03.470368 24662 ts_tablet_manager.cc:1918] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:40:03.472357 24661 ts_tablet_manager.cc:1918] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:40:03.472491 24661 log.cc:1196] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/wal/wals/8b32133f6264493780a62ac750260f02
I20260810 03:40:03.472800 24662 log.cc:1196] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/8b32133f6264493780a62ac750260f02
I20260810 03:40:03.472898 24661 ts_tablet_manager.cc:1939] T 8b32133f6264493780a62ac750260f02 P ba4e73928c5a48efa9d906fe8aa8ba4c: Deleting consensus metadata
I20260810 03:40:03.472949 24663 ts_tablet_manager.cc:1918] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:40:03.473111 24663 log.cc:1196] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/8b32133f6264493780a62ac750260f02
I20260810 03:40:03.473284 24662 ts_tablet_manager.cc:1939] T 8b32133f6264493780a62ac750260f02 P 6b8d906fb1cb4ecdb5f846746c90aa2d: Deleting consensus metadata
I20260810 03:40:03.473466 24663 ts_tablet_manager.cc:1939] T 8b32133f6264493780a62ac750260f02 P 6cf0d4b649f94dad9c7eb89fb2d2f604: Deleting consensus metadata
I20260810 03:40:03.473999 24019 catalog_manager.cc:4841] TS ba4e73928c5a48efa9d906fe8aa8ba4c (127.21.175.65:40223): tablet 8b32133f6264493780a62ac750260f02 (table test-table [id=adaac25ec3a540ce9e27690f9558bbe2]) successfully deleted
I20260810 03:40:03.474129 24018 catalog_manager.cc:4841] TS 6b8d906fb1cb4ecdb5f846746c90aa2d (127.21.175.66:34281): tablet 8b32133f6264493780a62ac750260f02 (table test-table [id=adaac25ec3a540ce9e27690f9558bbe2]) successfully deleted
I20260810 03:40:03.474416 24019 catalog_manager.cc:4841] TS 6cf0d4b649f94dad9c7eb89fb2d2f604 (127.21.175.67:40151): tablet 8b32133f6264493780a62ac750260f02 (table test-table [id=adaac25ec3a540ce9e27690f9558bbe2]) successfully deleted
Aug 10 03:40:03 dist-test-slave-ss4n krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333203, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:40:03 dist-test-slave-ss4n krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333203, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:03.498989 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:03.489849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57398 (local address 127.21.175.126:42539)
0810 03:40:03.490137 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:03.490144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:03.490161 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:40:03.490209 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:03.490211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:03.490256 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:03.490338 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:03.491096 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.491649 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.492447 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.492713 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.492792 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:03.495452 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:03.495455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:03.495457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:03.495483 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:03.497111 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:03.497634 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:03.497638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:03.497640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:03.497731 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:03.498031 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:03.498034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:03.498036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:03.498240 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:03.498379 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:03.498663 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 03:40:03.498800 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":85,"threads_started":1}
I20260810 03:40:03.501780 24033 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:40:03.502210 24033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:40:03.507637 24330 tablet_service.cc:1467] Processing CreateTablet for tablet af69c3a527f649f18ae3bc9beb0c6bd1 (DEFAULT_TABLE table=test-table [id=bc2faee06cbc4bbf8b38c7558d55c88d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:03.507690 24478 tablet_service.cc:1467] Processing CreateTablet for tablet af69c3a527f649f18ae3bc9beb0c6bd1 (DEFAULT_TABLE table=test-table [id=bc2faee06cbc4bbf8b38c7558d55c88d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:03.507653 24179 tablet_service.cc:1467] Processing CreateTablet for tablet af69c3a527f649f18ae3bc9beb0c6bd1 (DEFAULT_TABLE table=test-table [id=bc2faee06cbc4bbf8b38c7558d55c88d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:03.507977 24330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af69c3a527f649f18ae3bc9beb0c6bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:03.507977 24478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af69c3a527f649f18ae3bc9beb0c6bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:03.508008 24179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af69c3a527f649f18ae3bc9beb0c6bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:03.510764 24673 tablet_bootstrap.cc:492] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d: Bootstrap starting.
I20260810 03:40:03.510990 24675 tablet_bootstrap.cc:492] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c: Bootstrap starting.
I20260810 03:40:03.511750 24673 tablet_bootstrap.cc:654] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:03.512015 24675 tablet_bootstrap.cc:654] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:03.512413 24674 tablet_bootstrap.cc:492] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604: Bootstrap starting.
I20260810 03:40:03.513063 24673 tablet_bootstrap.cc:492] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d: No bootstrap required, opened a new log
I20260810 03:40:03.513157 24673 ts_tablet_manager.cc:1397] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:40:03.513378 24674 tablet_bootstrap.cc:654] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:03.513406 24675 tablet_bootstrap.cc:492] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c: No bootstrap required, opened a new log
I20260810 03:40:03.513553 24675 ts_tablet_manager.cc:1397] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:03.513713 24673 raft_consensus.cc:348] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.513834 24673 raft_consensus.cc:374] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:03.513870 24673 raft_consensus.cc:729] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8d906fb1cb4ecdb5f846746c90aa2d, State: Initialized, Role: FOLLOWER
I20260810 03:40:03.514026 24673 consensus_queue.cc:260] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.514360 24673 ts_tablet_manager.cc:1428] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:40:03.514497 24675 raft_consensus.cc:348] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.514616 24675 raft_consensus.cc:374] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:03.514631 24674 tablet_bootstrap.cc:492] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604: No bootstrap required, opened a new log
I20260810 03:40:03.514653 24675 raft_consensus.cc:729] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba4e73928c5a48efa9d906fe8aa8ba4c, State: Initialized, Role: FOLLOWER
I20260810 03:40:03.514729 24674 ts_tablet_manager.cc:1397] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:40:03.514838 24675 consensus_queue.cc:260] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.515205 24674 raft_consensus.cc:348] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.515295 24675 ts_tablet_manager.cc:1428] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:40:03.515322 24674 raft_consensus.cc:374] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:03.515355 24674 raft_consensus.cc:729] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf0d4b649f94dad9c7eb89fb2d2f604, State: Initialized, Role: FOLLOWER
I20260810 03:40:03.515488 24674 consensus_queue.cc:260] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.515892 24674 ts_tablet_manager.cc:1428] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:40:03.753407 24681 raft_consensus.cc:482] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:03.753650 24681 raft_consensus.cc:504] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.754854 24681 leader_election.cc:288] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba4e73928c5a48efa9d906fe8aa8ba4c (127.21.175.65:40223), 6b8d906fb1cb4ecdb5f846746c90aa2d (127.21.175.66:34281)
I20260810 03:40:03.756037 24680 raft_consensus.cc:482] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:03.756176 24680 raft_consensus.cc:504] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.758049 24680 leader_election.cc:288] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b8d906fb1cb4ecdb5f846746c90aa2d (127.21.175.66:34281), 6cf0d4b649f94dad9c7eb89fb2d2f604 (127.21.175.67:40151)
I20260810 03:40:03.761117 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:03.755119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:34281 (local address 127.21.175.67:50319)
0810 03:40:03.755539 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:03.755556 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:03.755695 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:03.756461 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:03.756465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:03.756485 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:03.756614 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:03.756621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.757846 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.757849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:03.759304 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:03.759317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.760442 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.760446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:03.760657 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:03.760721 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 03:40:03.760793 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 03:40:03.760910 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":153,"spinlock_wait_cycles":24832,"thread_start_us":163,"threads_started":1}
I20260810 03:40:03.761417 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:03.755197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:56411 (local address 127.21.175.65:40223)
0810 03:40:03.755653 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:03.755664 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:03.755682 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:40:03.755836 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:03.755841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:03.755898 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:03.756005 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:03.756496 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.757418 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.759467 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.760183 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.760308 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:03.760371 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:03.760931 (+ 560us) server_negotiation.cc:299] Negotiation successful
0810 03:40:03.761028 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":372,"thread_start_us":119,"threads_started":1}
I20260810 03:40:03.761688 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:03.755235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:50319 (local address 127.21.175.66:34281)
0810 03:40:03.755582 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:03.755624 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:03.755645 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:40:03.756194 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:03.756198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:03.756254 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:03.756342 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:03.756773 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.757713 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.759467 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.760176 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.760309 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:03.760371 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:03.761390 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0810 03:40:03.761547 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":96,"threads_started":1}
I20260810 03:40:03.762274 24683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:03.755104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:40223 (local address 127.21.175.67:56411)
0810 03:40:03.755539 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:03.755556 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:03.755695 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:03.756064 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:03.756068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:03.756091 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:03.756293 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:03.756305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.757641 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.757644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:03.759305 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:03.759317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.760442 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.760446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:03.760638 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:03.760711 (+ 73us) client_negotiation.cc:769] Sending connection context
0810 03:40:03.760793 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 03:40:03.762080 (+ 1287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":1279,"spinlock_wait_cycles":15232,"thread_start_us":183,"threads_started":1}
I20260810 03:40:03.762192 24353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af69c3a527f649f18ae3bc9beb0c6bd1" candidate_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" is_pre_election: true
I20260810 03:40:03.762493 24353 raft_consensus.cc:2454] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cf0d4b649f94dad9c7eb89fb2d2f604 in term 0.
I20260810 03:40:03.762876 24204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af69c3a527f649f18ae3bc9beb0c6bd1" candidate_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" is_pre_election: true
I20260810 03:40:03.763056 24204 raft_consensus.cc:2454] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cf0d4b649f94dad9c7eb89fb2d2f604 in term 0.
I20260810 03:40:03.763057 24428 leader_election.cc:302] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8d906fb1cb4ecdb5f846746c90aa2d, 6cf0d4b649f94dad9c7eb89fb2d2f604; no voters:
I20260810 03:40:03.763110 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:03.757669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:33313 (local address 127.21.175.67:40151)
0810 03:40:03.758219 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:03.758227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:03.758663 (+ 436us) server_negotiation.cc:407] Connection header received
0810 03:40:03.758834 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:03.758837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:03.758881 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:03.758939 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:03.759386 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.760147 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.761626 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.762267 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.762364 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:03.762410 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:03.762791 (+ 381us) server_negotiation.cc:299] Negotiation successful
0810 03:40:03.762872 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":98,"threads_started":1}
I20260810 03:40:03.763377 24681 raft_consensus.cc:2781] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:40:03.763455 24681 raft_consensus.cc:482] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:40:03.763514 24681 raft_consensus.cc:3037] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:03.763973 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:03.757915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:54153 (local address 127.21.175.66:34281)
0810 03:40:03.758369 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:03.758378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:03.758391 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:40:03.758532 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:03.758535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:03.758583 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:03.758676 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:03.759062 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.759862 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.761632 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.763220 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.763345 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:03.763397 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:03.763754 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 03:40:03.763850 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":85,"threads_started":1}
I20260810 03:40:03.763986 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:03.757533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:34281 (local address 127.21.175.65:54153)
0810 03:40:03.758280 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:03.758296 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:03.758400 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:03.758784 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:03.758789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:03.758809 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:03.758936 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:03.758943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.759992 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.759995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:03.761457 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:03.761469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.763464 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.763470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:03.763595 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:03.763620 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:40:03.763705 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 03:40:03.763791 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":70,"thread_start_us":108,"threads_started":1}
I20260810 03:40:03.763985 24686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:03.757854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:40151 (local address 127.21.175.65:33313)
0810 03:40:03.758581 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:03.758594 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:03.758724 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:03.759107 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:03.759109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:03.759128 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:03.759239 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:03.759245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.760292 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.760295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:03.761458 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:03.761469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.762537 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.762541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:03.762641 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:03.762667 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:40:03.762751 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 03:40:03.762865 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":35,"thread_start_us":81,"threads_started":1}
I20260810 03:40:03.764925 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af69c3a527f649f18ae3bc9beb0c6bd1" candidate_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" is_pre_election: true
I20260810 03:40:03.765017 24681 raft_consensus.cc:504] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.765208 24500 raft_consensus.cc:2379] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ba4e73928c5a48efa9d906fe8aa8ba4c in current term 1: Already voted for candidate 6cf0d4b649f94dad9c7eb89fb2d2f604 in this term.
I20260810 03:40:03.765410 24681 leader_election.cc:288] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [CANDIDATE]: Term 1 election: Requested vote from peers ba4e73928c5a48efa9d906fe8aa8ba4c (127.21.175.65:40223), 6b8d906fb1cb4ecdb5f846746c90aa2d (127.21.175.66:34281)
I20260810 03:40:03.765626 24353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af69c3a527f649f18ae3bc9beb0c6bd1" candidate_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" is_pre_election: true
I20260810 03:40:03.765754 24353 raft_consensus.cc:2454] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba4e73928c5a48efa9d906fe8aa8ba4c in term 0.
I20260810 03:40:03.765844 24352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af69c3a527f649f18ae3bc9beb0c6bd1" candidate_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d"
I20260810 03:40:03.765967 24352 raft_consensus.cc:3037] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:03.766059 24129 leader_election.cc:302] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b8d906fb1cb4ecdb5f846746c90aa2d, ba4e73928c5a48efa9d906fe8aa8ba4c; no voters: 6cf0d4b649f94dad9c7eb89fb2d2f604
I20260810 03:40:03.766350 24680 raft_consensus.cc:2781] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:40:03.766417 24680 raft_consensus.cc:482] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:40:03.766455 24680 raft_consensus.cc:3037] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:03.766652 24204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af69c3a527f649f18ae3bc9beb0c6bd1" candidate_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c"
I20260810 03:40:03.767256 24352 raft_consensus.cc:2454] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cf0d4b649f94dad9c7eb89fb2d2f604 in term 1.
I20260810 03:40:03.767521 24680 raft_consensus.cc:504] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.767642 24428 leader_election.cc:302] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8d906fb1cb4ecdb5f846746c90aa2d, 6cf0d4b649f94dad9c7eb89fb2d2f604; no voters:
I20260810 03:40:03.767812 24681 raft_consensus.cc:2781] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:03.767860 24204 raft_consensus.cc:2379] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6cf0d4b649f94dad9c7eb89fb2d2f604 in current term 1: Already voted for candidate ba4e73928c5a48efa9d906fe8aa8ba4c in this term.
I20260810 03:40:03.768095 24681 raft_consensus.cc:686] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 LEADER]: Becoming Leader. State: Replica: 6cf0d4b649f94dad9c7eb89fb2d2f604, State: Running, Role: LEADER
I20260810 03:40:03.768187 24680 leader_election.cc:288] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [CANDIDATE]: Term 1 election: Requested vote from peers 6b8d906fb1cb4ecdb5f846746c90aa2d (127.21.175.66:34281), 6cf0d4b649f94dad9c7eb89fb2d2f604 (127.21.175.67:40151)
I20260810 03:40:03.768371 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af69c3a527f649f18ae3bc9beb0c6bd1" candidate_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604"
I20260810 03:40:03.768410 24681 consensus_queue.cc:237] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.768774 24352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af69c3a527f649f18ae3bc9beb0c6bd1" candidate_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d"
I20260810 03:40:03.768930 24352 raft_consensus.cc:2379] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ba4e73928c5a48efa9d906fe8aa8ba4c in current term 1: Already voted for candidate 6cf0d4b649f94dad9c7eb89fb2d2f604 in this term.
I20260810 03:40:03.769274 24129 leader_election.cc:302] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba4e73928c5a48efa9d906fe8aa8ba4c; no voters: 6b8d906fb1cb4ecdb5f846746c90aa2d, 6cf0d4b649f94dad9c7eb89fb2d2f604
I20260810 03:40:03.769469 24680 raft_consensus.cc:2726] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 03:40:03.771323 24033 catalog_manager.cc:5495] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cf0d4b649f94dad9c7eb89fb2d2f604 (127.21.175.67). New cstate: current_term: 1 leader_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } health_report { overall_health: HEALTHY } } }
I20260810 03:40:03.819908 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:03.815900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.21.175.67:40151)
0810 03:40:03.816027 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:03.816031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:03.816123 (+ 92us) server_negotiation.cc:407] Connection header received
0810 03:40:03.816333 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:03.816336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:03.816393 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:03.816515 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:40:03.816793 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.817318 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.818282 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.818535 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.818654 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:03.818965 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:40:03.819471 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:40:03.819605 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:03.819674 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 03:40:03.819732 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:40:03.830971 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:03.827426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:51973 (local address 127.21.175.66:34281)
0810 03:40:03.827583 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:03.827605 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:03.827749 (+ 144us) server_negotiation.cc:407] Connection header received
0810 03:40:03.827919 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:03.827922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:03.827976 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:03.828062 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:40:03.828340 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.828865 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.829847 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.830072 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.830194 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:03.830500 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:40:03.830577 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:40:03.830663 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:03.830755 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 03:40:03.830810 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:40:03.831002 24695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:03.827274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:34281 (local address 127.21.175.67:51973)
0810 03:40:03.827662 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:03.827674 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:03.827810 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:03.828076 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:03.828078 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:03.828094 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:40:03.828228 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:03.828233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.828982 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.828987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:03.829724 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:03.829734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.830208 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.830212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:03.830306 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:03.830352 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:40:03.830670 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:40:03.830679 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 03:40:03.830744 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 03:40:03.830828 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":122,"threads_started":1}
I20260810 03:40:03.838268 24204 raft_consensus.cc:1261] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Refusing update from remote peer 6cf0d4b649f94dad9c7eb89fb2d2f604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:03.838373 24352 raft_consensus.cc:1261] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 FOLLOWER]: Refusing update from remote peer 6cf0d4b649f94dad9c7eb89fb2d2f604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:03.838797 24690 consensus_queue.cc:1035] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:03.839026 24681 consensus_queue.cc:1035] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:03.856936 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:03.853329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:40151 (local address 127.21.175.66:60529)
0810 03:40:03.853699 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:03.853715 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:03.853825 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:03.854107 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:03.854110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:03.854118 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:40:03.854227 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:03.854233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.854981 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.854986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:03.855627 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:03.855635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.856160 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.856164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:03.856263 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:03.856298 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:40:03.856620 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:40:03.856626 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 03:40:03.856708 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 03:40:03.856803 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260810 03:40:03.856940 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:03.853396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:60529 (local address 127.21.175.67:40151)
0810 03:40:03.853537 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:03.853542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:03.853800 (+ 258us) server_negotiation.cc:407] Connection header received
0810 03:40:03.853938 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:03.853941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:03.853997 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:03.854086 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:40:03.854343 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.854857 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.855779 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.856031 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.856138 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:03.856422 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:40:03.856510 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:40:03.856591 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:03.856738 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 03:40:03.856800 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:40:03.881376 24032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:03.881599 24032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:03.883847 24032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0: Sending DeleteTablet for 3 replicas of tablet af69c3a527f649f18ae3bc9beb0c6bd1
I20260810 03:40:03.884637 24179 tablet_service.cc:1514] Processing DeleteTablet for tablet af69c3a527f649f18ae3bc9beb0c6bd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:03 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:52594
I20260810 03:40:03.884853 24661 tablet_replica.cc:330] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c: stopping tablet replica
I20260810 03:40:03.884974 24661 raft_consensus.cc:2229] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:03.885047 24330 tablet_service.cc:1514] Processing DeleteTablet for tablet af69c3a527f649f18ae3bc9beb0c6bd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:03 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:39834
I20260810 03:40:03.885123 24661 raft_consensus.cc:2258] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:03.885257 24662 tablet_replica.cc:330] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d: stopping tablet replica
I20260810 03:40:03.885392 24662 raft_consensus.cc:2229] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:03.885442 24478 tablet_service.cc:1514] Processing DeleteTablet for tablet af69c3a527f649f18ae3bc9beb0c6bd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:03 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:46788
I20260810 03:40:03.885561 24662 raft_consensus.cc:2258] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:03.885730 24663 tablet_replica.cc:330] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604: stopping tablet replica
I20260810 03:40:03.885846 24663 raft_consensus.cc:2229] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:40:03.886143 24663 raft_consensus.cc:2258] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:03.886314 24662 ts_tablet_manager.cc:1905] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:03.886803 24663 ts_tablet_manager.cc:1905] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:03.886821 24032 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:40:03.887280 24032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:40:03.888635 24661 ts_tablet_manager.cc:1905] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:03.889335 24663 ts_tablet_manager.cc:1918] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:40:03.889335 24662 ts_tablet_manager.cc:1918] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:40:03.889407 24662 log.cc:1196] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/af69c3a527f649f18ae3bc9beb0c6bd1
I20260810 03:40:03.889407 24663 log.cc:1196] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/af69c3a527f649f18ae3bc9beb0c6bd1
I20260810 03:40:03.889727 24663 ts_tablet_manager.cc:1939] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6cf0d4b649f94dad9c7eb89fb2d2f604: Deleting consensus metadata
I20260810 03:40:03.890488 24019 catalog_manager.cc:4841] TS 6cf0d4b649f94dad9c7eb89fb2d2f604 (127.21.175.67:40151): tablet af69c3a527f649f18ae3bc9beb0c6bd1 (table test-table [id=bc2faee06cbc4bbf8b38c7558d55c88d]) successfully deleted
I20260810 03:40:03.891188 24661 ts_tablet_manager.cc:1918] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:40:03.891276 24661 log.cc:1196] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-0/wal/wals/af69c3a527f649f18ae3bc9beb0c6bd1
I20260810 03:40:03.891528 24661 ts_tablet_manager.cc:1939] T af69c3a527f649f18ae3bc9beb0c6bd1 P ba4e73928c5a48efa9d906fe8aa8ba4c: Deleting consensus metadata
I20260810 03:40:03.891871 24662 ts_tablet_manager.cc:1939] T af69c3a527f649f18ae3bc9beb0c6bd1 P 6b8d906fb1cb4ecdb5f846746c90aa2d: Deleting consensus metadata
I20260810 03:40:03.892390 24019 catalog_manager.cc:4841] TS ba4e73928c5a48efa9d906fe8aa8ba4c (127.21.175.65:40223): tablet af69c3a527f649f18ae3bc9beb0c6bd1 (table test-table [id=bc2faee06cbc4bbf8b38c7558d55c88d]) successfully deleted
I20260810 03:40:03.892565 24018 catalog_manager.cc:4841] TS 6b8d906fb1cb4ecdb5f846746c90aa2d (127.21.175.66:34281): tablet af69c3a527f649f18ae3bc9beb0c6bd1 (table test-table [id=bc2faee06cbc4bbf8b38c7558d55c88d]) successfully deleted
I20260810 03:40:03.894932 24179 tablet_service.cc:1467] Processing CreateTablet for tablet 043a9979c73f4c9ab9aab953d19dab9e (DEFAULT_TABLE table=test-table [id=efe01fd7c4a243168bdb5abb2086948b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:03.894950 24330 tablet_service.cc:1467] Processing CreateTablet for tablet 043a9979c73f4c9ab9aab953d19dab9e (DEFAULT_TABLE table=test-table [id=efe01fd7c4a243168bdb5abb2086948b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:03.895211 24179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043a9979c73f4c9ab9aab953d19dab9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:03.895259 24330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043a9979c73f4c9ab9aab953d19dab9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:03.897326 24673 tablet_bootstrap.cc:492] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d: Bootstrap starting.
I20260810 03:40:03.897689 24675 tablet_bootstrap.cc:492] T 043a9979c73f4c9ab9aab953d19dab9e P ba4e73928c5a48efa9d906fe8aa8ba4c: Bootstrap starting.
I20260810 03:40:03.898408 24673 tablet_bootstrap.cc:654] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:03.898609 24675 tablet_bootstrap.cc:654] T 043a9979c73f4c9ab9aab953d19dab9e P ba4e73928c5a48efa9d906fe8aa8ba4c: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:03.899370 24478 tablet_service.cc:1467] Processing CreateTablet for tablet 043a9979c73f4c9ab9aab953d19dab9e (DEFAULT_TABLE table=test-table [id=efe01fd7c4a243168bdb5abb2086948b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:03.899657 24478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043a9979c73f4c9ab9aab953d19dab9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:03.899773 24673 tablet_bootstrap.cc:492] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d: No bootstrap required, opened a new log
I20260810 03:40:03.899900 24673 ts_tablet_manager.cc:1397] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:03.900316 24673 raft_consensus.cc:348] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.900461 24673 raft_consensus.cc:374] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:03.900513 24673 raft_consensus.cc:729] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8d906fb1cb4ecdb5f846746c90aa2d, State: Initialized, Role: FOLLOWER
I20260810 03:40:03.900774 24673 consensus_queue.cc:260] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.901221 24673 ts_tablet_manager.cc:1428] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 03:40:03.902024 24674 tablet_bootstrap.cc:492] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604: Bootstrap starting.
I20260810 03:40:03.902994 24674 tablet_bootstrap.cc:654] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:03.904292 24675 tablet_bootstrap.cc:492] T 043a9979c73f4c9ab9aab953d19dab9e P ba4e73928c5a48efa9d906fe8aa8ba4c: No bootstrap required, opened a new log
I20260810 03:40:03.904392 24675 ts_tablet_manager.cc:1397] T 043a9979c73f4c9ab9aab953d19dab9e P ba4e73928c5a48efa9d906fe8aa8ba4c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260810 03:40:03.904871 24675 raft_consensus.cc:348] T 043a9979c73f4c9ab9aab953d19dab9e P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.905002 24675 raft_consensus.cc:374] T 043a9979c73f4c9ab9aab953d19dab9e P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:03.905035 24675 raft_consensus.cc:729] T 043a9979c73f4c9ab9aab953d19dab9e P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba4e73928c5a48efa9d906fe8aa8ba4c, State: Initialized, Role: FOLLOWER
I20260810 03:40:03.905160 24675 consensus_queue.cc:260] T 043a9979c73f4c9ab9aab953d19dab9e P ba4e73928c5a48efa9d906fe8aa8ba4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.905454 24675 ts_tablet_manager.cc:1428] T 043a9979c73f4c9ab9aab953d19dab9e P ba4e73928c5a48efa9d906fe8aa8ba4c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 03:40:03.906556 24674 tablet_bootstrap.cc:492] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604: No bootstrap required, opened a new log
I20260810 03:40:03.906658 24674 ts_tablet_manager.cc:1397] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 03:40:03.907083 24674 raft_consensus.cc:348] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.907200 24674 raft_consensus.cc:374] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:03.907231 24674 raft_consensus.cc:729] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf0d4b649f94dad9c7eb89fb2d2f604, State: Initialized, Role: FOLLOWER
I20260810 03:40:03.907351 24674 consensus_queue.cc:260] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.907770 24674 ts_tablet_manager.cc:1428] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:40:03.959005 24648 raft_consensus.cc:482] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:03.959174 24648 raft_consensus.cc:504] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.959631 24648 leader_election.cc:288] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba4e73928c5a48efa9d906fe8aa8ba4c (127.21.175.65:40223), 6cf0d4b649f94dad9c7eb89fb2d2f604 (127.21.175.67:40151)
I20260810 03:40:03.960189 24204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043a9979c73f4c9ab9aab953d19dab9e" candidate_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" is_pre_election: true
I20260810 03:40:03.960227 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043a9979c73f4c9ab9aab953d19dab9e" candidate_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" is_pre_election: true
I20260810 03:40:03.960371 24204 raft_consensus.cc:2454] T 043a9979c73f4c9ab9aab953d19dab9e P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8d906fb1cb4ecdb5f846746c90aa2d in term 0.
I20260810 03:40:03.960371 24500 raft_consensus.cc:2454] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8d906fb1cb4ecdb5f846746c90aa2d in term 0.
I20260810 03:40:03.960740 24282 leader_election.cc:302] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8d906fb1cb4ecdb5f846746c90aa2d, 6cf0d4b649f94dad9c7eb89fb2d2f604; no voters:
I20260810 03:40:03.960953 24648 raft_consensus.cc:2781] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:40:03.961028 24648 raft_consensus.cc:482] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:40:03.961068 24648 raft_consensus.cc:3037] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:03.962411 24648 raft_consensus.cc:504] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.962790 24648 leader_election.cc:288] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [CANDIDATE]: Term 1 election: Requested vote from peers ba4e73928c5a48efa9d906fe8aa8ba4c (127.21.175.65:40223), 6cf0d4b649f94dad9c7eb89fb2d2f604 (127.21.175.67:40151)
I20260810 03:40:03.963260 24204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043a9979c73f4c9ab9aab953d19dab9e" candidate_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c"
I20260810 03:40:03.963320 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043a9979c73f4c9ab9aab953d19dab9e" candidate_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604"
I20260810 03:40:03.963413 24204 raft_consensus.cc:3037] T 043a9979c73f4c9ab9aab953d19dab9e P ba4e73928c5a48efa9d906fe8aa8ba4c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:03.963446 24500 raft_consensus.cc:3037] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:03.964393 24204 raft_consensus.cc:2454] T 043a9979c73f4c9ab9aab953d19dab9e P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8d906fb1cb4ecdb5f846746c90aa2d in term 1.
I20260810 03:40:03.964551 24500 raft_consensus.cc:2454] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8d906fb1cb4ecdb5f846746c90aa2d in term 1.
I20260810 03:40:03.964740 24282 leader_election.cc:302] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8d906fb1cb4ecdb5f846746c90aa2d, ba4e73928c5a48efa9d906fe8aa8ba4c; no voters:
I20260810 03:40:03.964987 24648 raft_consensus.cc:2781] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:03.965107 24648 raft_consensus.cc:686] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 LEADER]: Becoming Leader. State: Replica: 6b8d906fb1cb4ecdb5f846746c90aa2d, State: Running, Role: LEADER
I20260810 03:40:03.965257 24648 consensus_queue.cc:237] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } }
I20260810 03:40:03.967348 24033 catalog_manager.cc:5495] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b8d906fb1cb4ecdb5f846746c90aa2d (127.21.175.66). New cstate: current_term: 1 leader_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b8d906fb1cb4ecdb5f846746c90aa2d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 } health_report { overall_health: UNKNOWN } } }
I20260810 03:40:03.998814 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:03.995302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.21.175.66:34281)
0810 03:40:03.995416 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:03.995425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:03.995438 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:40:03.995569 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:03.995571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:03.995621 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:03.995688 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:40:03.995969 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.996482 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.997713 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:03.997944 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:03.998062 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:03.998330 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:40:03.998405 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:40:03.998461 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:03.998595 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 03:40:03.998646 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 03:40:04.006039 24500 raft_consensus.cc:1261] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Refusing update from remote peer 6b8d906fb1cb4ecdb5f846746c90aa2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:04.006068 24204 raft_consensus.cc:1261] T 043a9979c73f4c9ab9aab953d19dab9e P ba4e73928c5a48efa9d906fe8aa8ba4c [term 1 FOLLOWER]: Refusing update from remote peer 6b8d906fb1cb4ecdb5f846746c90aa2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:04.006480 24638 consensus_queue.cc:1035] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba4e73928c5a48efa9d906fe8aa8ba4c" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:04.006671 24642 consensus_queue.cc:1035] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cf0d4b649f94dad9c7eb89fb2d2f604" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 03:40:04.023955 24640 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 03:40:05.592518 24032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:05.592756 24032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:05.595520 24032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ceef1a53d4249de9dbee1a00b7c91d0: Sending DeleteTablet for 3 replicas of tablet 043a9979c73f4c9ab9aab953d19dab9e
I20260810 03:40:05.596311 24179 tablet_service.cc:1514] Processing DeleteTablet for tablet 043a9979c73f4c9ab9aab953d19dab9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:05 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:52594
I20260810 03:40:05.596311 24330 tablet_service.cc:1514] Processing DeleteTablet for tablet 043a9979c73f4c9ab9aab953d19dab9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:05 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:39834
I20260810 03:40:05.596683 24478 tablet_service.cc:1514] Processing DeleteTablet for tablet 043a9979c73f4c9ab9aab953d19dab9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:05 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:46788
I20260810 03:40:05.596751 24733 tablet_replica.cc:330] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d: stopping tablet replica
I20260810 03:40:05.596714 24732 tablet_replica.cc:330] T 043a9979c73f4c9ab9aab953d19dab9e P ba4e73928c5a48efa9d906fe8aa8ba4c: stopping tablet replica
I20260810 03:40:05.596803 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 24112
I20260810 03:40:05.599534 24734 tablet_replica.cc:330] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604: stopping tablet replica
I20260810 03:40:05.599684 24734 raft_consensus.cc:2229] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:05.599823 24734 raft_consensus.cc:2258] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:05.596889 24733 raft_consensus.cc:2229] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:40:05.600167 24733 raft_consensus.cc:2258] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:05.600647 24734 ts_tablet_manager.cc:1905] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:05.601172 24733 ts_tablet_manager.cc:1905] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:05.603965 24733 ts_tablet_manager.cc:1918] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:40:05.604079 24733 log.cc:1196] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/043a9979c73f4c9ab9aab953d19dab9e
I20260810 03:40:05.604349 24733 ts_tablet_manager.cc:1939] T 043a9979c73f4c9ab9aab953d19dab9e P 6b8d906fb1cb4ecdb5f846746c90aa2d: Deleting consensus metadata
I20260810 03:40:05.604725 24734 ts_tablet_manager.cc:1918] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:40:05.604818 24734 log.cc:1196] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/043a9979c73f4c9ab9aab953d19dab9e
I20260810 03:40:05.605101 24734 ts_tablet_manager.cc:1939] T 043a9979c73f4c9ab9aab953d19dab9e P 6cf0d4b649f94dad9c7eb89fb2d2f604: Deleting consensus metadata
I20260810 03:40:05.605715 24018 catalog_manager.cc:4841] TS 6b8d906fb1cb4ecdb5f846746c90aa2d (127.21.175.66:34281): tablet 043a9979c73f4c9ab9aab953d19dab9e (table test-table [id=efe01fd7c4a243168bdb5abb2086948b]) successfully deleted
I20260810 03:40:05.606262 24019 catalog_manager.cc:4841] TS 6cf0d4b649f94dad9c7eb89fb2d2f604 (127.21.175.67:40151): tablet 043a9979c73f4c9ab9aab953d19dab9e (table test-table [id=efe01fd7c4a243168bdb5abb2086948b]) successfully deleted
W20260810 03:40:05.608649 24019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.175.65:40223 (error 108)
I20260810 03:40:05.608938 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 24263
I20260810 03:40:05.612421 24736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:05.611372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:40223 (local address 127.0.0.1:49196)
0810 03:40:05.612141 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:05.612266 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":158,"threads_started":1}
W20260810 03:40:05.612599 24019 catalog_manager.cc:4584] TS ba4e73928c5a48efa9d906fe8aa8ba4c (127.21.175.65:40223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 043a9979c73f4c9ab9aab953d19dab9e: Network error: Client connection negotiation failed: client connection to 127.21.175.65:40223: connect: Connection refused (error 111)
I20260810 03:40:05.620980 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 24412
I20260810 03:40:05.631794 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 24002
2026-08-10T03:40:05Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7269 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:40:05 dist-test-slave-ss4n krb5kdc[24739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:40:05 dist-test-slave-ss4n krb5kdc[24739](info): set up 2 sockets
Aug 10 03:40:05 dist-test-slave-ss4n krb5kdc[24739](info): commencing operation
krb5kdc: starting...
W20260810 03:40:06.805073 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:40:06 dist-test-slave-ss4n krb5kdc[24739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:40:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:40:06.945286 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:35503
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:35503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:07.063723 24753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:07.064087 24753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:07.064180 24753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:07.068420 24753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:07.068537 24753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:07.068576 24753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:07.068630 24753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:07.068687 24753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:40:07.074344 24753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:35503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:35503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:07.075502 24753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:07.076618 24753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:07.085865 24762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:07.085915 24759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:07.086030 24760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:07.085981 24753 server_base.cc:1034] running on GCE node
I20260810 03:40:07.086859 24753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:07.088012 24753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:07.089213 24753 hybrid_clock.cc:648] HybridClock initialized: now 1786333207089206 us; error 32 us; skew 500 ppm
Aug 10 03:40:07 dist-test-slave-ss4n krb5kdc[24739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333207, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:07.092190 24753 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:40:07.093277 24753 webserver.cc:460] Webserver started at http://127.21.175.126:44363/ using document root <none> and password file <none>
I20260810 03:40:07.093915 24753 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:07.093981 24753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:07.094221 24753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:07.096343 24753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "6170750a3fb940c5a8ddc5494978cfd1"
format_stamp: "Formatted at 2026-08-10 03:40:07 on dist-test-slave-ss4n"
server_key: "ea12fd1318338c828d6bceae00bb1914"
server_key_iv: "42e0a723b3ab4dc55105636eb86cd660"
server_key_version: "encryptionkey@0"
I20260810 03:40:07.096952 24753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "6170750a3fb940c5a8ddc5494978cfd1"
format_stamp: "Formatted at 2026-08-10 03:40:07 on dist-test-slave-ss4n"
server_key: "ea12fd1318338c828d6bceae00bb1914"
server_key_iv: "42e0a723b3ab4dc55105636eb86cd660"
server_key_version: "encryptionkey@0"
I20260810 03:40:07.100960 24753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:40:07.103740 24769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:07.105122 24753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 03:40:07.105261 24753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "6170750a3fb940c5a8ddc5494978cfd1"
format_stamp: "Formatted at 2026-08-10 03:40:07 on dist-test-slave-ss4n"
server_key: "ea12fd1318338c828d6bceae00bb1914"
server_key_iv: "42e0a723b3ab4dc55105636eb86cd660"
server_key_version: "encryptionkey@0"
I20260810 03:40:07.105424 24753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:07.132098 24753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:07.135455 24753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:07.135689 24753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:07.143290 24753 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:35503
I20260810 03:40:07.143329 24825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:35503 every 8 connection(s)
I20260810 03:40:07.144328 24753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:07.147253 24826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:07.152890 24826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1: Bootstrap starting.
I20260810 03:40:07.153332 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 24753
I20260810 03:40:07.153455 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:40:07.153740 22205 external_mini_cluster.cc:1428] Setting key c038d7393219a6a8a741e4842a91333e
I20260810 03:40:07.155599 24826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:07.156347 24826 log.cc:824] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:07.158748 24826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1: No bootstrap required, opened a new log
I20260810 03:40:07.163431 24826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6170750a3fb940c5a8ddc5494978cfd1" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35503 } }
I20260810 03:40:07.163720 24826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Aug 10 03:40:07 dist-test-slave-ss4n krb5kdc[24739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:07.163797 24826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6170750a3fb940c5a8ddc5494978cfd1, State: Initialized, Role: FOLLOWER
I20260810 03:40:07.164261 24826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6170750a3fb940c5a8ddc5494978cfd1" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35503 } }
I20260810 03:40:07.164410 24826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:07.164477 24826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:07.164616 24826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:07.165841 24826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6170750a3fb940c5a8ddc5494978cfd1" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35503 } }
I20260810 03:40:07.166198 24826 leader_election.cc:302] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6170750a3fb940c5a8ddc5494978cfd1; no voters:
I20260810 03:40:07.166576 24826 leader_election.cc:288] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:40:07.166651 24831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:07.166889 24831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [term 1 LEADER]: Becoming Leader. State: Replica: 6170750a3fb940c5a8ddc5494978cfd1, State: Running, Role: LEADER
I20260810 03:40:07.167277 24831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6170750a3fb940c5a8ddc5494978cfd1" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35503 } }
I20260810 03:40:07.168000 24826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:07.169036 24833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6170750a3fb940c5a8ddc5494978cfd1. Latest consensus state: current_term: 1 leader_uuid: "6170750a3fb940c5a8ddc5494978cfd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6170750a3fb940c5a8ddc5494978cfd1" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35503 } } }
I20260810 03:40:07.169451 24833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:07.169323 24832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6170750a3fb940c5a8ddc5494978cfd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6170750a3fb940c5a8ddc5494978cfd1" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35503 } } }
I20260810 03:40:07.169781 24832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1 [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:07.169880 24840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:07.170117 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:07.155187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60838 (local address 127.21.175.126:35503)
0810 03:40:07.155609 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:07.155625 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:07.155687 (+ 62us) server_negotiation.cc:407] Connection header received
0810 03:40:07.156340 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:07.156356 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:07.156790 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:07.157180 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:07.158188 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:07.159308 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:07.160492 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:07.160971 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:07.161090 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:07.164400 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:07.164407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:07.164424 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:07.164461 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:07.166429 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:07.166992 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:07.166999 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:07.167005 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:07.167178 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:07.167521 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:07.167523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:07.167524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:07.167984 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:07.168202 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:07.168466 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 03:40:07.168930 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":135,"threads_started":1}
I20260810 03:40:07.174497 24840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:07.180452 24840 catalog_manager.cc:1347] Generated new cluster ID: 440ab1ce938c463dbee00026b1e724b4
I20260810 03:40:07.180539 24840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:07.192310 24840 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:40:07.193392 24840 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:07.198376 24840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6170750a3fb940c5a8ddc5494978cfd1: Generated new TSK 0
I20260810 03:40:07.199257 24840 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:40:07.238095 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35503
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:07.358314 24852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:07.358603 24852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:07.358683 24852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:07.362782 24852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:07.362852 24852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:07.362913 24852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:40:07.368505 24852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:07.369771 24852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:07.370874 24852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:07.381438 24857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:07.381374 24858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:07.381371 24860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:07.381426 24852 server_base.cc:1034] running on GCE node
I20260810 03:40:07.382162 24852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:07.382877 24852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:07.384137 24852 hybrid_clock.cc:648] HybridClock initialized: now 1786333207384090 us; error 51 us; skew 500 ppm
Aug 10 03:40:07 dist-test-slave-ss4n krb5kdc[24739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333207, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:07.387544 24852 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:40:07.388577 24852 webserver.cc:460] Webserver started at http://127.21.175.65:37255/ using document root <none> and password file <none>
I20260810 03:40:07.389163 24852 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:07.389256 24852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:07.389561 24852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:07.391662 24852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "c7c04e09ab7047f08d59ddd27f27849a"
format_stamp: "Formatted at 2026-08-10 03:40:07 on dist-test-slave-ss4n"
server_key: "5fd6937ea5a9d7dd9d18dcf859074348"
server_key_iv: "7595d191891cb8e046c66cbe75bbbedb"
server_key_version: "encryptionkey@0"
I20260810 03:40:07.392259 24852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "c7c04e09ab7047f08d59ddd27f27849a"
format_stamp: "Formatted at 2026-08-10 03:40:07 on dist-test-slave-ss4n"
server_key: "5fd6937ea5a9d7dd9d18dcf859074348"
server_key_iv: "7595d191891cb8e046c66cbe75bbbedb"
server_key_version: "encryptionkey@0"
I20260810 03:40:07.396554 24852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260810 03:40:07.399426 24867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:07.400789 24852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:07.400938 24852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "c7c04e09ab7047f08d59ddd27f27849a"
format_stamp: "Formatted at 2026-08-10 03:40:07 on dist-test-slave-ss4n"
server_key: "5fd6937ea5a9d7dd9d18dcf859074348"
server_key_iv: "7595d191891cb8e046c66cbe75bbbedb"
server_key_version: "encryptionkey@0"
I20260810 03:40:07.401067 24852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:07.422348 24852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:07.426007 24852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:07.426275 24852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:07.426959 24852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:07.427884 24852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:07.427959 24852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:07.428038 24852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:07.428090 24852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:07.437574 24852 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:45027
I20260810 03:40:07.437603 24986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:45027 every 8 connection(s)
I20260810 03:40:07.438603 24852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:40:07.446341 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 24852
I20260810 03:40:07.446488 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:40:07.446776 22205 external_mini_cluster.cc:1428] Setting key 75fcb9548f83fdf7b732f6d2732d6962
Aug 10 03:40:07 dist-test-slave-ss4n krb5kdc[24739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333207, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:07.454191 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:07.440575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:36423 (local address 127.21.175.126:35503)
0810 03:40:07.440737 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:07.440740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:07.441808 (+ 1068us) server_negotiation.cc:407] Connection header received
0810 03:40:07.443253 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:07.443257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:07.443318 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:07.443432 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:07.444688 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:07.445224 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:07.446519 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:07.446805 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:07.446932 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:07.450101 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:07.450104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:07.450106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:07.450130 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:07.451651 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:07.452326 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:07.452333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:07.452337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:07.452442 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:07.452817 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:07.452823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:07.452828 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:07.453133 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:07.453246 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:07.453870 (+ 624us) server_negotiation.cc:299] Negotiation successful
0810 03:40:07.454015 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:40:07.455132 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:07.440962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:35503 (local address 127.21.175.65:36423)
0810 03:40:07.441637 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:07.441687 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:07.443008 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:07.443623 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:07.443630 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:07.444021 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:07.444528 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:07.444541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:07.445366 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:07.445370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:07.446377 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:07.446388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:07.446996 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:07.447001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:07.447209 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:07.447924 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:07.447941 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:07.449904 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:07.451786 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:07.451796 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:07.451818 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:07.452194 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:07.452552 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:07.452555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:07.452557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:07.452709 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:07.453243 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:07.453249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:07.453595 (+ 346us) client_negotiation.cc:769] Sending connection context
0810 03:40:07.453810 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 03:40:07.454208 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":168,"threads_started":1}
I20260810 03:40:07.456563 24987 heartbeater.cc:347] Connected to a master server at 127.21.175.126:35503
I20260810 03:40:07.456833 24987 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:07.457403 24987 heartbeater.cc:511] Master 127.21.175.126:35503 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
I20260810 03:40:07.459309 24788 ts_manager.cc:194] Registered new tserver with Master: c7c04e09ab7047f08d59ddd27f27849a (127.21.175.65:45027)
I20260810 03:40:07.460563 24788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:36423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:40:07.481350 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35503
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:07.608031 24992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:07.608335 24992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:07.608381 24992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:07.612859 24992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:07.612941 24992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:07.613003 24992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:40:07.618474 24992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:07.619691 24992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:07.620827 24992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:07.631122 25000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:07.631119 24998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:07.631119 24997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:07.631163 24992 server_base.cc:1034] running on GCE node
I20260810 03:40:07.631795 24992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:07.632403 24992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:07.633561 24992 hybrid_clock.cc:648] HybridClock initialized: now 1786333207633524 us; error 46 us; skew 500 ppm
Aug 10 03:40:07 dist-test-slave-ss4n krb5kdc[24739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333207, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:07.636495 24992 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:40:07.637633 24992 webserver.cc:460] Webserver started at http://127.21.175.66:39659/ using document root <none> and password file <none>
I20260810 03:40:07.638284 24992 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:07.638388 24992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:07.638638 24992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:07.640905 24992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "52e5ecbb4f124da593fd016bde91fcde"
format_stamp: "Formatted at 2026-08-10 03:40:07 on dist-test-slave-ss4n"
server_key: "977dc837da88966aba0d6a47dc871d39"
server_key_iv: "4b225ed0e4e5cf697c8fff25e763777b"
server_key_version: "encryptionkey@0"
I20260810 03:40:07.641606 24992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "52e5ecbb4f124da593fd016bde91fcde"
format_stamp: "Formatted at 2026-08-10 03:40:07 on dist-test-slave-ss4n"
server_key: "977dc837da88966aba0d6a47dc871d39"
server_key_iv: "4b225ed0e4e5cf697c8fff25e763777b"
server_key_version: "encryptionkey@0"
I20260810 03:40:07.645781 24992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:40:07.648439 25007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:07.649892 24992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:07.650033 24992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "52e5ecbb4f124da593fd016bde91fcde"
format_stamp: "Formatted at 2026-08-10 03:40:07 on dist-test-slave-ss4n"
server_key: "977dc837da88966aba0d6a47dc871d39"
server_key_iv: "4b225ed0e4e5cf697c8fff25e763777b"
server_key_version: "encryptionkey@0"
I20260810 03:40:07.650161 24992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:07.663579 24992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:07.667337 24992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:07.667586 24992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:07.668323 24992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:07.669459 24992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:07.669561 24992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:07.669643 24992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:07.669674 24992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:07.679943 24992 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:37811
I20260810 03:40:07.679975 25127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:37811 every 8 connection(s)
I20260810 03:40:07.680905 24992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:40:07.690202 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 24992
I20260810 03:40:07.690317 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:40:07.690642 22205 external_mini_cluster.cc:1428] Setting key bd57e21df0a2bc409027406df6ad3713
Aug 10 03:40:07 dist-test-slave-ss4n krb5kdc[24739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333207, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:07.696707 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:07.683085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:51885 (local address 127.21.175.126:35503)
0810 03:40:07.683261 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:07.683266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:07.684205 (+ 939us) server_negotiation.cc:407] Connection header received
0810 03:40:07.685781 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:07.685784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:07.685834 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:07.685922 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:07.687313 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:07.687915 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:07.689113 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:07.689349 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:07.689441 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:07.692725 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:07.692728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:07.692730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:07.692754 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:07.694358 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:07.695033 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:07.695036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:07.695039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:07.695142 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:07.695519 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:07.695525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:07.695530 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:07.695740 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:07.695849 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:07.696412 (+ 563us) server_negotiation.cc:299] Negotiation successful
0810 03:40:07.696541 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:40:07.697748 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:07.683498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:35503 (local address 127.21.175.66:51885)
0810 03:40:07.684028 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:07.684077 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:07.685445 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:07.686153 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:07.686160 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:07.686576 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:07.687148 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:07.687163 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:07.688054 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:07.688057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:07.688973 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:07.688983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:07.689560 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:07.689565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:07.689796 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:07.690557 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:07.690575 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:07.692504 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:07.694494 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:07.694504 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:07.694538 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:07.694900 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:07.695247 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:07.695250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:07.695252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:07.695411 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:07.695864 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:07.695874 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:07.696148 (+ 274us) client_negotiation.cc:769] Sending connection context
0810 03:40:07.696355 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 03:40:07.696734 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":110,"threads_started":1}
I20260810 03:40:07.699324 25128 heartbeater.cc:347] Connected to a master server at 127.21.175.126:35503
I20260810 03:40:07.699599 25128 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:07.700119 25128 heartbeater.cc:511] Master 127.21.175.126:35503 requested a full tablet report, sending...
I20260810 03:40:07.701256 24788 ts_manager.cc:194] Registered new tserver with Master: 52e5ecbb4f124da593fd016bde91fcde (127.21.175.66:37811)
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
I20260810 03:40:07.702033 24788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:51885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:40:07.725143 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35503
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:07.857667 25134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:07.858026 25134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:07.858075 25134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:07.862421 25134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:07.862541 25134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:07.862612 25134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:40:07.868285 25134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:07.869560 25134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:07.870700 25134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:07.881014 25142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:07.881215 25134 server_base.cc:1034] running on GCE node
W20260810 03:40:07.881038 25139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:07.881409 25140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:07.881918 25134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:07.882879 25134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:07.884130 25134 hybrid_clock.cc:648] HybridClock initialized: now 1786333207884100 us; error 32 us; skew 500 ppm
Aug 10 03:40:07 dist-test-slave-ss4n krb5kdc[24739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333207, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:07.887616 25134 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:40:07.888724 25134 webserver.cc:460] Webserver started at http://127.21.175.67:43475/ using document root <none> and password file <none>
I20260810 03:40:07.889356 25134 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:07.889457 25134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:07.889734 25134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:07.892117 25134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "22117112bd8c4d7d97190089d5f89514"
format_stamp: "Formatted at 2026-08-10 03:40:07 on dist-test-slave-ss4n"
server_key: "ebaac68375e60455bdbab67bab9c8263"
server_key_iv: "1b7fd983faf1ba76cbd8c642703e9674"
server_key_version: "encryptionkey@0"
I20260810 03:40:07.892683 25134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "22117112bd8c4d7d97190089d5f89514"
format_stamp: "Formatted at 2026-08-10 03:40:07 on dist-test-slave-ss4n"
server_key: "ebaac68375e60455bdbab67bab9c8263"
server_key_iv: "1b7fd983faf1ba76cbd8c642703e9674"
server_key_version: "encryptionkey@0"
I20260810 03:40:07.896950 25134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260810 03:40:07.899847 25149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:07.901258 25134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:07.901389 25134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "22117112bd8c4d7d97190089d5f89514"
format_stamp: "Formatted at 2026-08-10 03:40:07 on dist-test-slave-ss4n"
server_key: "ebaac68375e60455bdbab67bab9c8263"
server_key_iv: "1b7fd983faf1ba76cbd8c642703e9674"
server_key_version: "encryptionkey@0"
I20260810 03:40:07.901574 25134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:07.912576 25134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:07.917573 25134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:07.917832 25134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:07.918708 25134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:07.919991 25134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:07.920061 25134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:07.920117 25134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:07.920152 25134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:07.929700 25134 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:42881
I20260810 03:40:07.929783 25268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:42881 every 8 connection(s)
I20260810 03:40:07.930658 25134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:40:07.933884 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 25134
I20260810 03:40:07.934015 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:40:07.934372 22205 external_mini_cluster.cc:1428] Setting key c180eca95fcc2e7f97909c5181b6a849
Aug 10 03:40:07 dist-test-slave-ss4n krb5kdc[24739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333207, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:07.946344 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:07.932557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:47151 (local address 127.21.175.126:35503)
0810 03:40:07.932711 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:07.932715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:07.934308 (+ 1593us) server_negotiation.cc:407] Connection header received
0810 03:40:07.935451 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:07.935454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:07.935510 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:07.935614 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:07.937044 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:07.937647 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:07.938851 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:07.939079 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:07.939202 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:07.942033 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:07.942037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:07.942039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:07.942065 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:07.943647 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:07.944229 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:07.944238 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:07.944239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:07.944323 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:07.944698 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:07.944702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:07.944704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:07.944911 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:07.945042 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:07.945997 (+ 955us) server_negotiation.cc:299] Negotiation successful
0810 03:40:07.946168 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 03:40:07.947320 25272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:07.933063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:35503 (local address 127.21.175.67:47151)
0810 03:40:07.934111 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:07.934177 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:07.935233 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:07.935815 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:07.935823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:07.936293 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:07.936905 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:07.936923 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:07.937775 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:07.937780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:07.938717 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:07.938726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:07.939252 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:07.939256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:07.939437 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:07.940174 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:07.940192 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:07.941834 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:07.943784 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:07.943791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:07.943810 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:07.944106 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:07.944421 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:07.944426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:07.944428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:07.944587 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:07.945041 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:07.945050 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:07.945709 (+ 659us) client_negotiation.cc:769] Sending connection context
0810 03:40:07.945950 (+ 241us) client_negotiation.cc:240] Negotiation successful
0810 03:40:07.946336 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":633,"threads_started":1}
I20260810 03:40:07.948544 25269 heartbeater.cc:347] Connected to a master server at 127.21.175.126:35503
I20260810 03:40:07.948824 25269 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:07.949352 25269 heartbeater.cc:511] Master 127.21.175.126:35503 requested a full tablet report, sending...
I20260810 03:40:07.950474 24787 ts_manager.cc:194] Registered new tserver with Master: 22117112bd8c4d7d97190089d5f89514 (127.21.175.67:42881)
I20260810 03:40:07.951113 24787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:47151
I20260810 03:40:07.961265 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 03:40:07.977331 24829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:07.975842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60842 (local address 127.21.175.126:35503)
0810 03:40:07.976020 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:07.976024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:07.976075 (+ 51us) server_negotiation.cc:407] Connection header received
0810 03:40:07.976206 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:07.976209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:07.976266 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:07.976340 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:07.977176 (+ 836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60842: BlockingRecv error: recv got EOF from 127.0.0.1:60842 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:40:07.979030 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 24852
I20260810 03:40:07.987118 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 24992
I20260810 03:40:07.994885 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 25134
I20260810 03:40:08.004990 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 24753
2026-08-10T03:40:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2372 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:40:08 dist-test-slave-ss4n krb5kdc[25283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:40:08 dist-test-slave-ss4n krb5kdc[25283](info): set up 2 sockets
Aug 10 03:40:08 dist-test-slave-ss4n krb5kdc[25283](info): commencing operation
krb5kdc: starting...
W20260810 03:40:09.188306 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260810 03:40:09.339026 22205 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.151s user 0.002s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260810 03:40:09.572438 22205 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.233s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260810 03:40:09.737916 22205 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.165s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/krb5kdc/test-user.keytab.
W20260810 03:40:10.308822 22205 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.571s user 0.000s sys 0.002s
Aug 10 03:40:10 dist-test-slave-ss4n krb5kdc[25283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:40:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
W20260810 03:40:10.574582 22205 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.175.126: real 0.236s user 0.000s sys 0.004s
I20260810 03:40:10.575222 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:42629
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:42629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:10.696949 25298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:10.697340 25298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:10.697424 25298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:10.701846 25298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:10.701972 25298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:10.702013 25298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:10.702042 25298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:10.702073 25298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:40:10.707896 25298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:42629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:42629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:10.709194 25298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:10.710532 25298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:10.719825 25303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:10.719832 25306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:10.719830 25304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:10.720276 25298 server_base.cc:1034] running on GCE node
I20260810 03:40:10.720976 25298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:10.722306 25298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:10.723553 25298 hybrid_clock.cc:648] HybridClock initialized: now 1786333210723509 us; error 54 us; skew 500 ppm
Aug 10 03:40:10 dist-test-slave-ss4n krb5kdc[25283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333210, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:10.732661 25298 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:40:10.733819 25298 webserver.cc:460] Webserver started at http://127.21.175.126:35129/ using document root <none> and password file <none>
I20260810 03:40:10.734496 25298 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:10.734597 25298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:10.734848 25298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:10.737112 25298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "e0aba61ef6e24e2789251fc99b475a0a"
format_stamp: "Formatted at 2026-08-10 03:40:10 on dist-test-slave-ss4n"
server_key: "2f071878607e6c6b660a882e1d297e84"
server_key_iv: "ef532d34c510db35f0723bbf8488600e"
server_key_version: "encryptionkey@0"
I20260810 03:40:10.737795 25298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "e0aba61ef6e24e2789251fc99b475a0a"
format_stamp: "Formatted at 2026-08-10 03:40:10 on dist-test-slave-ss4n"
server_key: "2f071878607e6c6b660a882e1d297e84"
server_key_iv: "ef532d34c510db35f0723bbf8488600e"
server_key_version: "encryptionkey@0"
I20260810 03:40:10.741938 25298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:40:10.744711 25314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:10.746183 25298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:10.746335 25298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "e0aba61ef6e24e2789251fc99b475a0a"
format_stamp: "Formatted at 2026-08-10 03:40:10 on dist-test-slave-ss4n"
server_key: "2f071878607e6c6b660a882e1d297e84"
server_key_iv: "ef532d34c510db35f0723bbf8488600e"
server_key_version: "encryptionkey@0"
I20260810 03:40:10.746464 25298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:10.765662 25298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:10.769451 25298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:10.769723 25298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:10.777561 25298 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:42629
I20260810 03:40:10.777854 25369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:42629 every 8 connection(s)
I20260810 03:40:10.778743 25298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:10.781689 25370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:10.783660 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 25298
I20260810 03:40:10.783757 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:40:10.784070 22205 external_mini_cluster.cc:1428] Setting key 052d32524a5446414c20a204370354ae
I20260810 03:40:10.789197 25370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a: Bootstrap starting.
I20260810 03:40:10.792122 25370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:10.793114 25370 log.cc:824] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:40:10 dist-test-slave-ss4n krb5kdc[25283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:10.795866 25370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a: No bootstrap required, opened a new log
I20260810 03:40:10.800982 25370 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aba61ef6e24e2789251fc99b475a0a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42629 } }
I20260810 03:40:10.801318 25370 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:10.801373 25370 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0aba61ef6e24e2789251fc99b475a0a, State: Initialized, Role: FOLLOWER
I20260810 03:40:10.801812 25373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:10.785776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.21.175.126:42629)
0810 03:40:10.786273 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:10.786301 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:10.786345 (+ 44us) server_negotiation.cc:407] Connection header received
0810 03:40:10.787203 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:10.787231 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:10.787702 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:10.788120 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:10.788957 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:10.790456 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:10.791428 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:10.792157 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:10.792334 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:10.795904 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:10.795913 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:10.795934 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:10.795970 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:10.798097 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:10.798671 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:10.798680 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:10.798690 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:10.798875 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:10.799177 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:10.799180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:10.799182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:10.799783 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:10.799999 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:10.800282 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 03:40:10.800647 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":132,"threads_started":1}
I20260810 03:40:10.801915 25370 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aba61ef6e24e2789251fc99b475a0a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42629 } }
I20260810 03:40:10.802086 25370 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:10.802202 25370 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:10.802310 25370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:10.803534 25370 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aba61ef6e24e2789251fc99b475a0a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42629 } }
I20260810 03:40:10.803930 25370 leader_election.cc:302] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0aba61ef6e24e2789251fc99b475a0a; no voters:
I20260810 03:40:10.804387 25370 leader_election.cc:288] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:40:10.804512 25375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:10.804816 25375 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [term 1 LEADER]: Becoming Leader. State: Replica: e0aba61ef6e24e2789251fc99b475a0a, State: Running, Role: LEADER
I20260810 03:40:10.805145 25375 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aba61ef6e24e2789251fc99b475a0a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42629 } }
I20260810 03:40:10.805207 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:40:10.805907 25370 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:10.807085 25377 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0aba61ef6e24e2789251fc99b475a0a. Latest consensus state: current_term: 1 leader_uuid: "e0aba61ef6e24e2789251fc99b475a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aba61ef6e24e2789251fc99b475a0a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42629 } } }
I20260810 03:40:10.807231 25377 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:10.807073 25376 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0aba61ef6e24e2789251fc99b475a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aba61ef6e24e2789251fc99b475a0a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42629 } } }
I20260810 03:40:10.807345 25376 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:10.807668 25385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:10.810739 25385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:10.819585 25385 catalog_manager.cc:1347] Generated new cluster ID: b348cd7df6aa4ab58f5bc36dc389ae9f
I20260810 03:40:10.819674 25385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:10.828640 25373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:10.822080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.21.175.126:42629)
0810 03:40:10.822198 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:10.822202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:10.822483 (+ 281us) server_negotiation.cc:407] Connection header received
0810 03:40:10.822611 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:10.822615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:10.822678 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:10.822767 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:10.823360 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:10.824206 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:10.825407 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:10.825822 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:10.825955 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:10.827004 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:10.827008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:10.827011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:10.827643 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:10.827685 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:10.827709 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:10.827777 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:10.827977 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 03:40:10.828484 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 03:40:10.829437 25385 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:40:10.830611 25385 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:10.839529 25385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0aba61ef6e24e2789251fc99b475a0a: Generated new TSK 0
I20260810 03:40:10.840480 25385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:40:10.851574 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 25298
2026-08-10T03:40:10Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2844 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:40:10 dist-test-slave-ss4n krb5kdc[25404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:40:10 dist-test-slave-ss4n krb5kdc[25404](info): set up 2 sockets
Aug 10 03:40:10 dist-test-slave-ss4n krb5kdc[25404](info): commencing operation
krb5kdc: starting...
W20260810 03:40:11.993726 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:40:12 dist-test-slave-ss4n krb5kdc[25404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:40:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:40:12.138290 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:37459
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:37459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:12.254876 25418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:12.255242 25418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:12.255329 25418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:12.259502 25418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:12.259605 25418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:12.259640 25418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:12.259678 25418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:12.259712 25418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:40:12.265246 25418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:37459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:37459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:12.266440 25418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:12.267531 25418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:12.276412 25426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:12.276412 25424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:12.276412 25423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:12.276943 25418 server_base.cc:1034] running on GCE node
I20260810 03:40:12.277606 25418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:12.278839 25418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:12.280049 25418 hybrid_clock.cc:648] HybridClock initialized: now 1786333212280041 us; error 33 us; skew 500 ppm
Aug 10 03:40:12 dist-test-slave-ss4n krb5kdc[25404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333212, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:12.283205 25418 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:40:12.284348 25418 webserver.cc:460] Webserver started at http://127.21.175.126:46085/ using document root <none> and password file <none>
I20260810 03:40:12.284960 25418 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:12.285032 25418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:12.285245 25418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:12.287818 25418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "9762fe749f7646c2bfc4c6fd1514ca7e"
format_stamp: "Formatted at 2026-08-10 03:40:12 on dist-test-slave-ss4n"
server_key: "83227c8408a605870744cc8c801a223a"
server_key_iv: "425cb1c7a8544a018a9b4ee306c7b331"
server_key_version: "encryptionkey@0"
I20260810 03:40:12.288421 25418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "9762fe749f7646c2bfc4c6fd1514ca7e"
format_stamp: "Formatted at 2026-08-10 03:40:12 on dist-test-slave-ss4n"
server_key: "83227c8408a605870744cc8c801a223a"
server_key_iv: "425cb1c7a8544a018a9b4ee306c7b331"
server_key_version: "encryptionkey@0"
I20260810 03:40:12.292865 25418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:40:12.295985 25434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:12.297410 25418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:12.297650 25418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "9762fe749f7646c2bfc4c6fd1514ca7e"
format_stamp: "Formatted at 2026-08-10 03:40:12 on dist-test-slave-ss4n"
server_key: "83227c8408a605870744cc8c801a223a"
server_key_iv: "425cb1c7a8544a018a9b4ee306c7b331"
server_key_version: "encryptionkey@0"
I20260810 03:40:12.297783 25418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:12.310580 25418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:12.314394 25418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:12.314659 25418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:12.323324 25418 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:37459
I20260810 03:40:12.323643 25490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:37459 every 8 connection(s)
I20260810 03:40:12.324494 25418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:12.326162 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 25418
I20260810 03:40:12.326318 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:40:12.326686 22205 external_mini_cluster.cc:1428] Setting key a90856ae228c2fad2d6ee6a6aa300810
I20260810 03:40:12.328014 25491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:12.335814 25491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e: Bootstrap starting.
Aug 10 03:40:12 dist-test-slave-ss4n krb5kdc[25404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:12.338486 25491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:12.339509 25491 log.cc:824] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:12.342065 25491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e: No bootstrap required, opened a new log
I20260810 03:40:12.344091 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:12.328209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.21.175.126:37459)
0810 03:40:12.328700 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:12.328721 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:12.328765 (+ 44us) server_negotiation.cc:407] Connection header received
0810 03:40:12.329678 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:12.329703 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:12.330186 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:12.330585 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:12.331397 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:12.332925 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:12.334006 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:12.334790 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:12.334957 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:12.338235 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:12.338241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:12.338256 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:12.338292 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:12.340381 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:12.340931 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:12.340937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:12.340942 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:12.341078 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:12.341437 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:12.341440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:12.341441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:12.342043 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:12.342314 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:12.342666 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 03:40:12.343064 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":128,"threads_started":1}
I20260810 03:40:12.345706 25491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9762fe749f7646c2bfc4c6fd1514ca7e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37459 } }
I20260810 03:40:12.346012 25491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:12.346071 25491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9762fe749f7646c2bfc4c6fd1514ca7e, State: Initialized, Role: FOLLOWER
I20260810 03:40:12.346587 25491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9762fe749f7646c2bfc4c6fd1514ca7e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37459 } }
I20260810 03:40:12.346794 25491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:12.346879 25491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:12.346992 25491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:12.348083 25491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9762fe749f7646c2bfc4c6fd1514ca7e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37459 } }
I20260810 03:40:12.348455 25491 leader_election.cc:302] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9762fe749f7646c2bfc4c6fd1514ca7e; no voters:
I20260810 03:40:12.348875 25491 leader_election.cc:288] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:40:12.349020 25496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:12.349313 25496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [term 1 LEADER]: Becoming Leader. State: Replica: 9762fe749f7646c2bfc4c6fd1514ca7e, State: Running, Role: LEADER
I20260810 03:40:12.349761 25496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9762fe749f7646c2bfc4c6fd1514ca7e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37459 } }
I20260810 03:40:12.350268 25491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:12.352159 25498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9762fe749f7646c2bfc4c6fd1514ca7e. Latest consensus state: current_term: 1 leader_uuid: "9762fe749f7646c2bfc4c6fd1514ca7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9762fe749f7646c2bfc4c6fd1514ca7e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37459 } } }
I20260810 03:40:12.352322 25498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:12.353039 25497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9762fe749f7646c2bfc4c6fd1514ca7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9762fe749f7646c2bfc4c6fd1514ca7e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37459 } } }
I20260810 03:40:12.353130 25506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:12.353229 25497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:12.356230 25506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:12.362773 25506 catalog_manager.cc:1347] Generated new cluster ID: 40fba48eb5f540269af2cfc502116813
I20260810 03:40:12.362864 25506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:12.378793 25506 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:40:12.380384 25506 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:12.389132 25506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9762fe749f7646c2bfc4c6fd1514ca7e: Generated new TSK 0
I20260810 03:40:12.390218 25506 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:40:12.430217 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:37459
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:12.554949 25519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:12.555297 25519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:12.555377 25519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:12.559530 25519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:12.559623 25519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:12.559718 25519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:40:12.565155 25519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:37459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:12.566359 25519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:12.567431 25519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:12.577725 25524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:12.577697 25525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:12.577704 25527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:12.577912 25519 server_base.cc:1034] running on GCE node
I20260810 03:40:12.578382 25519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:12.579097 25519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:12.580307 25519 hybrid_clock.cc:648] HybridClock initialized: now 1786333212580275 us; error 44 us; skew 500 ppm
Aug 10 03:40:12 dist-test-slave-ss4n krb5kdc[25404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333212, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:12.583449 25519 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:40:12.584506 25519 webserver.cc:460] Webserver started at http://127.21.175.65:42267/ using document root <none> and password file <none>
I20260810 03:40:12.585110 25519 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:12.585206 25519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:12.585455 25519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:12.587687 25519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "82eeb163ec694f6597752b5f9632ebe4"
format_stamp: "Formatted at 2026-08-10 03:40:12 on dist-test-slave-ss4n"
server_key: "644f7b677d03f71a4100b34c3f6fed98"
server_key_iv: "91de0d822cf2e64a24a7275350b74aa1"
server_key_version: "encryptionkey@0"
I20260810 03:40:12.588299 25519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "82eeb163ec694f6597752b5f9632ebe4"
format_stamp: "Formatted at 2026-08-10 03:40:12 on dist-test-slave-ss4n"
server_key: "644f7b677d03f71a4100b34c3f6fed98"
server_key_iv: "91de0d822cf2e64a24a7275350b74aa1"
server_key_version: "encryptionkey@0"
I20260810 03:40:12.592331 25519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:40:12.595162 25535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:12.596599 25519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:40:12.596766 25519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "82eeb163ec694f6597752b5f9632ebe4"
format_stamp: "Formatted at 2026-08-10 03:40:12 on dist-test-slave-ss4n"
server_key: "644f7b677d03f71a4100b34c3f6fed98"
server_key_iv: "91de0d822cf2e64a24a7275350b74aa1"
server_key_version: "encryptionkey@0"
I20260810 03:40:12.596905 25519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:12.611912 25519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:12.615435 25519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:12.615662 25519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:12.616319 25519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:12.617174 25519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:12.617244 25519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:12.617316 25519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:12.617360 25519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:12.626540 25519 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:41425
I20260810 03:40:12.626569 25655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:41425 every 8 connection(s)
I20260810 03:40:12.627482 25519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:40:12.628471 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 25519
I20260810 03:40:12.628604 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:40:12.628893 22205 external_mini_cluster.cc:1428] Setting key 4e65514d5729dd306b2a99661545c7b2
Aug 10 03:40:12 dist-test-slave-ss4n krb5kdc[25404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333212, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
I20260810 03:40:12.643692 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:12.630304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:42585 (local address 127.21.175.126:37459)
0810 03:40:12.630442 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:12.630446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:12.631488 (+ 1042us) server_negotiation.cc:407] Connection header received
0810 03:40:12.632899 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:12.632902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:12.632969 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:12.633119 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:12.634446 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:12.634994 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:12.636202 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:12.636441 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:12.636552 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:12.639266 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:12.639269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:12.639271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:12.639297 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:12.640861 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:12.641673 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:12.641675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:12.641677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:12.641756 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:12.642206 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:12.642212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:12.642214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:12.642493 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:12.642609 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:12.643384 (+ 775us) server_negotiation.cc:299] Negotiation successful
0810 03:40:12.643547 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:40:12.645008 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:12.630800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:37459 (local address 127.21.175.65:42585)
0810 03:40:12.631306 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:12.631363 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:12.632675 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:12.633306 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:12.633313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:12.633789 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:12.634286 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:12.634300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:12.635141 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:12.635144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:12.636056 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:12.636066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:12.636613 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:12.636617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:12.636807 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:12.637456 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:12.637525 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:12.639093 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:12.641010 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:12.641016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:12.641031 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:12.641562 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:12.641846 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:12.641849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:12.641850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:12.642078 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:12.642637 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:12.642645 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:12.643032 (+ 387us) client_negotiation.cc:769] Sending connection context
0810 03:40:12.643288 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 03:40:12.643734 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":153,"threads_started":1}
I20260810 03:40:12.646525 25656 heartbeater.cc:347] Connected to a master server at 127.21.175.126:37459
I20260810 03:40:12.646816 25656 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:12.647369 25656 heartbeater.cc:511] Master 127.21.175.126:37459 requested a full tablet report, sending...
I20260810 03:40:12.649058 25452 ts_manager.cc:194] Registered new tserver with Master: 82eeb163ec694f6597752b5f9632ebe4 (127.21.175.65:41425)
I20260810 03:40:12.650408 25452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:42585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:40:12.661669 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:37459
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:12.780714 25661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:12.780998 25661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:12.781046 25661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:12.785238 25661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:12.785315 25661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:12.785375 25661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:40:12.790937 25661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:37459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:12.792083 25661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:12.793128 25661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:12.803695 25667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:12.803896 25661 server_base.cc:1034] running on GCE node
W20260810 03:40:12.803706 25668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:12.803679 25671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:12.804458 25661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:12.805127 25661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:12.806330 25661 hybrid_clock.cc:648] HybridClock initialized: now 1786333212806274 us; error 75 us; skew 500 ppm
Aug 10 03:40:12 dist-test-slave-ss4n krb5kdc[25404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333212, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:12.809361 25661 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:40:12.810424 25661 webserver.cc:460] Webserver started at http://127.21.175.66:45579/ using document root <none> and password file <none>
I20260810 03:40:12.810956 25661 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:12.811024 25661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:12.811195 25661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:12.813234 25661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "dab56952d64e4934be4296eeb0bbe774"
format_stamp: "Formatted at 2026-08-10 03:40:12 on dist-test-slave-ss4n"
server_key: "b70e71bd47190107a6af253d2db44ad2"
server_key_iv: "3ec186c5a610895bc9be3d5045aade2a"
server_key_version: "encryptionkey@0"
I20260810 03:40:12.813839 25661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "dab56952d64e4934be4296eeb0bbe774"
format_stamp: "Formatted at 2026-08-10 03:40:12 on dist-test-slave-ss4n"
server_key: "b70e71bd47190107a6af253d2db44ad2"
server_key_iv: "3ec186c5a610895bc9be3d5045aade2a"
server_key_version: "encryptionkey@0"
I20260810 03:40:12.817854 25661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:40:12.820381 25678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:12.821745 25661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:12.821877 25661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "dab56952d64e4934be4296eeb0bbe774"
format_stamp: "Formatted at 2026-08-10 03:40:12 on dist-test-slave-ss4n"
server_key: "b70e71bd47190107a6af253d2db44ad2"
server_key_iv: "3ec186c5a610895bc9be3d5045aade2a"
server_key_version: "encryptionkey@0"
I20260810 03:40:12.821995 25661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:12.841010 25661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:12.844762 25661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:12.844978 25661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:12.845666 25661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:12.846520 25661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:12.846592 25661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:12.846666 25661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:12.846719 25661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:12.856760 25661 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:33427
I20260810 03:40:12.856787 25796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:33427 every 8 connection(s)
I20260810 03:40:12.857786 25661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:40:12.859964 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 25661
I20260810 03:40:12.860085 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:40:12.860394 22205 external_mini_cluster.cc:1428] Setting key 9d245b976d332b2d8c850f17079e60f8
Aug 10 03:40:12 dist-test-slave-ss4n krb5kdc[25404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333212, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
I20260810 03:40:12.874409 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:12.859902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:33185 (local address 127.21.175.126:37459)
0810 03:40:12.860095 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:12.860100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:12.861576 (+ 1476us) server_negotiation.cc:407] Connection header received
0810 03:40:12.863059 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:12.863066 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:12.863135 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:12.863248 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:12.864603 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:12.865164 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:12.866410 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:12.866651 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:12.866766 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:12.869829 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:12.869837 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:12.869843 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:12.869879 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:12.871840 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:12.872552 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:12.872556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:12.872558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:12.872673 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:12.873094 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:12.873097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:12.873098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:12.873309 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:12.873396 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:12.874126 (+ 730us) server_negotiation.cc:299] Negotiation successful
0810 03:40:12.874254 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:40:12.875367 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:12.860456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:37459 (local address 127.21.175.66:33185)
0810 03:40:12.861116 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:12.861165 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:12.862798 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:12.863453 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:12.863460 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:12.863864 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:12.864437 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:12.864453 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:12.865297 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:12.865301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:12.866276 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:12.866286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:12.866811 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:12.866815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:12.866992 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:12.867638 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:12.867656 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:12.869610 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:12.872015 (+ 2405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:12.872021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:12.872037 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:12.872352 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:12.872811 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:12.872814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:12.872815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:12.872977 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:12.873429 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:12.873437 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:12.873831 (+ 394us) client_negotiation.cc:769] Sending connection context
0810 03:40:12.874010 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 03:40:12.874344 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":152,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
I20260810 03:40:12.876855 25797 heartbeater.cc:347] Connected to a master server at 127.21.175.126:37459
I20260810 03:40:12.877127 25797 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:12.877717 25797 heartbeater.cc:511] Master 127.21.175.126:37459 requested a full tablet report, sending...
I20260810 03:40:12.879009 25452 ts_manager.cc:194] Registered new tserver with Master: dab56952d64e4934be4296eeb0bbe774 (127.21.175.66:33427)
I20260810 03:40:12.879690 25452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:33185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:40:12.897917 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:37459
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:13.019997 25802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:13.020292 25802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:13.020339 25802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:13.024559 25802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:13.024634 25802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:13.024696 25802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:40:13.030352 25802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:37459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:13.031553 25802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:13.032666 25802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:40:13.043576 25802 server_base.cc:1034] running on GCE node
W20260810 03:40:13.043668 25808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:13.043536 25809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:13.043536 25811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:13.044409 25802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:13.045106 25802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:13.046303 25802 hybrid_clock.cc:648] HybridClock initialized: now 1786333213046238 us; error 79 us; skew 500 ppm
Aug 10 03:40:13 dist-test-slave-ss4n krb5kdc[25404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333213, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:13.049790 25802 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:40:13.051118 25802 webserver.cc:460] Webserver started at http://127.21.175.67:40827/ using document root <none> and password file <none>
I20260810 03:40:13.052021 25802 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:13.052161 25802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:13.052484 25802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:13.055089 25802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "50615fa686d845c1ba136d2dda1fedad"
format_stamp: "Formatted at 2026-08-10 03:40:13 on dist-test-slave-ss4n"
server_key: "14027686a82bff681d6a9396037bdb08"
server_key_iv: "40543546b4500729ca4137c1b8fc8f0e"
server_key_version: "encryptionkey@0"
I20260810 03:40:13.055806 25802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "50615fa686d845c1ba136d2dda1fedad"
format_stamp: "Formatted at 2026-08-10 03:40:13 on dist-test-slave-ss4n"
server_key: "14027686a82bff681d6a9396037bdb08"
server_key_iv: "40543546b4500729ca4137c1b8fc8f0e"
server_key_version: "encryptionkey@0"
I20260810 03:40:13.060158 25802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:40:13.063113 25820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:13.064424 25802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:40:13.064616 25802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "50615fa686d845c1ba136d2dda1fedad"
format_stamp: "Formatted at 2026-08-10 03:40:13 on dist-test-slave-ss4n"
server_key: "14027686a82bff681d6a9396037bdb08"
server_key_iv: "40543546b4500729ca4137c1b8fc8f0e"
server_key_version: "encryptionkey@0"
I20260810 03:40:13.064754 25802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:13.106454 25802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:13.110044 25802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:13.110267 25802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:13.110940 25802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:13.111825 25802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:13.111899 25802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:13.111972 25802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:13.112027 25802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:13.121876 25802 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:34313
I20260810 03:40:13.121901 25940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:34313 every 8 connection(s)
I20260810 03:40:13.122993 25802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:40:13.127243 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 25802
I20260810 03:40:13.127410 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:40:13.127789 22205 external_mini_cluster.cc:1428] Setting key 3e285cac8201d5423740b9bc2951f122
Aug 10 03:40:13 dist-test-slave-ss4n krb5kdc[25404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333213, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:13.139118 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:13.125155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:37697 (local address 127.21.175.126:37459)
0810 03:40:13.125383 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:13.125389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:13.126524 (+ 1135us) server_negotiation.cc:407] Connection header received
0810 03:40:13.128481 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:13.128485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:13.128548 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:13.128637 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:13.130553 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:13.131099 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:13.132338 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:13.132600 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:13.132687 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:13.135453 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:13.135456 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:13.135458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:13.135489 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:13.137064 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:13.137638 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:13.137641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:13.137642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:13.137724 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:13.138095 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:13.138097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:13.138098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:13.138283 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:13.138357 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:13.138879 (+ 522us) server_negotiation.cc:299] Negotiation successful
0810 03:40:13.138992 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:40:13.140065 25943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:13.125731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:37459 (local address 127.21.175.67:37697)
0810 03:40:13.126251 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:13.126300 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:13.128239 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:13.128991 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:13.129010 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:13.129564 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:13.130318 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:13.130340 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:13.131247 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:13.131250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:13.132206 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:13.132216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:13.132752 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:13.132755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:13.132907 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:13.133667 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:13.133684 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:13.135252 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:13.137186 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:13.137192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:13.137206 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:13.137469 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:13.137822 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:13.137825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:13.137826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:13.138000 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:13.138370 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:13.138375 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:13.138637 (+ 262us) client_negotiation.cc:769] Sending connection context
0810 03:40:13.138811 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 03:40:13.139103 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":141,"threads_started":1}
I20260810 03:40:13.141608 25941 heartbeater.cc:347] Connected to a master server at 127.21.175.126:37459
I20260810 03:40:13.141909 25941 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:13.142467 25941 heartbeater.cc:511] Master 127.21.175.126:37459 requested a full tablet report, sending...
I20260810 03:40:13.143565 25452 ts_manager.cc:194] Registered new tserver with Master: 50615fa686d845c1ba136d2dda1fedad (127.21.175.67:34313)
I20260810 03:40:13.144176 25452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:37697
I20260810 03:40:13.155194 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:40:13 dist-test-slave-ss4n krb5kdc[25404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333213, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 03:40:13 dist-test-slave-ss4n krb5kdc[25404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333213, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:13.185014 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:13.175903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37266 (local address 127.21.175.126:37459)
0810 03:40:13.176103 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:13.176107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:13.176185 (+ 78us) server_negotiation.cc:407] Connection header received
0810 03:40:13.176379 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:13.176383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:13.176435 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:13.176526 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:13.177290 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:13.177924 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:13.178820 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:13.179043 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:13.179141 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:13.181635 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:13.181639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:13.181641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:13.181667 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:13.183164 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:13.183681 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:13.183683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:13.183685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:13.183762 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:13.184044 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:13.184046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:13.184048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:13.184293 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:13.184400 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:13.184682 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 03:40:13.184782 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260810 03:40:13.185595 25452 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37266
I20260810 03:40:13.186671 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 25519
I20260810 03:40:13.197347 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 25661
I20260810 03:40:13.208192 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 25802
I20260810 03:40:13.216548 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 25418
2026-08-10T03:40:13Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2367 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:40:13 dist-test-slave-ss4n krb5kdc[25954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:40:13 dist-test-slave-ss4n krb5kdc[25954](info): set up 2 sockets
Aug 10 03:40:13 dist-test-slave-ss4n krb5kdc[25954](info): commencing operation
krb5kdc: starting...
W20260810 03:40:14.402937 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:40:14 dist-test-slave-ss4n krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:40:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:40:14.546105 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:35173
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:35173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:14.665126 25968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:14.665431 25968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:14.665549 25968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:14.669708 25968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:14.669781 25968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:14.669804 25968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:14.669826 25968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:14.669883 25968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:40:14.675465 25968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:35173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:35173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:14.676644 25968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:14.677806 25968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:14.687098 25974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:14.687116 25976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:14.687156 25968 server_base.cc:1034] running on GCE node
W20260810 03:40:14.687055 25973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:14.688064 25968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:14.689313 25968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:14.690580 25968 hybrid_clock.cc:648] HybridClock initialized: now 1786333214690563 us; error 58 us; skew 500 ppm
Aug 10 03:40:14 dist-test-slave-ss4n krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333214, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:14.693892 25968 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:40:14.695044 25968 webserver.cc:460] Webserver started at http://127.21.175.126:45153/ using document root <none> and password file <none>
I20260810 03:40:14.695922 25968 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:14.696024 25968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:14.696262 25968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:14.698590 25968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "644f1559c5724df3aeaac747f20ff790"
format_stamp: "Formatted at 2026-08-10 03:40:14 on dist-test-slave-ss4n"
server_key: "2b546c84b48e4ff72f54de46eb15e296"
server_key_iv: "ac2b72cfb2e7497278b1376c2fe4f411"
server_key_version: "encryptionkey@0"
I20260810 03:40:14.699159 25968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "644f1559c5724df3aeaac747f20ff790"
format_stamp: "Formatted at 2026-08-10 03:40:14 on dist-test-slave-ss4n"
server_key: "2b546c84b48e4ff72f54de46eb15e296"
server_key_iv: "ac2b72cfb2e7497278b1376c2fe4f411"
server_key_version: "encryptionkey@0"
I20260810 03:40:14.703913 25968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:40:14.706647 25984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:14.708069 25968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:14.708202 25968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "644f1559c5724df3aeaac747f20ff790"
format_stamp: "Formatted at 2026-08-10 03:40:14 on dist-test-slave-ss4n"
server_key: "2b546c84b48e4ff72f54de46eb15e296"
server_key_iv: "ac2b72cfb2e7497278b1376c2fe4f411"
server_key_version: "encryptionkey@0"
I20260810 03:40:14.708307 25968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:14.739751 25968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:14.743234 25968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:14.743407 25968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:14.751199 25968 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:35173
I20260810 03:40:14.751221 26039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:35173 every 8 connection(s)
I20260810 03:40:14.752244 25968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:14.754297 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 25968
I20260810 03:40:14.754410 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:40:14.754725 22205 external_mini_cluster.cc:1428] Setting key 017e46ae9ea465dd057ef46cc13fc8bc
I20260810 03:40:14.755334 26040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:14.761765 26040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790: Bootstrap starting.
Aug 10 03:40:14 dist-test-slave-ss4n krb5kdc[25954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:14.764297 26040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:14.765055 26040 log.cc:824] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:14.767062 26040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790: No bootstrap required, opened a new log
I20260810 03:40:14.769315 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:14.756213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.21.175.126:35173)
0810 03:40:14.756667 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:14.756682 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:14.756720 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:40:14.757390 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:14.757415 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:14.757853 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:14.758206 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:14.759063 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:14.760121 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:14.760989 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:14.761549 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:14.761722 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:14.764554 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:14.764559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:14.764573 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:14.764610 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:14.766429 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:14.766905 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:14.766909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:14.766914 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:14.767026 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:14.767331 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:14.767334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:14.767336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:14.767748 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:14.767953 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:14.768280 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 03:40:14.768618 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":144,"threads_started":1}
I20260810 03:40:14.771000 26040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644f1559c5724df3aeaac747f20ff790" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35173 } }
I20260810 03:40:14.771277 26040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:14.771328 26040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644f1559c5724df3aeaac747f20ff790, State: Initialized, Role: FOLLOWER
I20260810 03:40:14.771816 26040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644f1559c5724df3aeaac747f20ff790" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35173 } }
I20260810 03:40:14.771958 26040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:14.772006 26040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:14.772076 26040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:14.772943 26040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644f1559c5724df3aeaac747f20ff790" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35173 } }
I20260810 03:40:14.773238 26040 leader_election.cc:302] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 644f1559c5724df3aeaac747f20ff790; no voters:
I20260810 03:40:14.773552 26040 leader_election.cc:288] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:40:14.773658 26046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:14.773943 26046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [term 1 LEADER]: Becoming Leader. State: Replica: 644f1559c5724df3aeaac747f20ff790, State: Running, Role: LEADER
I20260810 03:40:14.774259 26046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644f1559c5724df3aeaac747f20ff790" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35173 } }
I20260810 03:40:14.774725 26040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:14.776005 26047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "644f1559c5724df3aeaac747f20ff790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644f1559c5724df3aeaac747f20ff790" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35173 } } }
I20260810 03:40:14.776139 26047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:14.775978 26048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 644f1559c5724df3aeaac747f20ff790. Latest consensus state: current_term: 1 leader_uuid: "644f1559c5724df3aeaac747f20ff790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644f1559c5724df3aeaac747f20ff790" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35173 } } }
I20260810 03:40:14.776283 26048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790 [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:14.776620 26055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:14.780445 26055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:14.786026 26055 catalog_manager.cc:1347] Generated new cluster ID: b69bd064df3d45188d09da45ee983e87
I20260810 03:40:14.786099 26055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:14.803432 26055 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:40:14.804792 26055 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:14.818064 26055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 644f1559c5724df3aeaac747f20ff790: Generated new TSK 0
I20260810 03:40:14.818956 26055 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:40:14.861660 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35173
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:14.994516 26067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:14.994879 26067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:14.994959 26067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:14.999533 26067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:14.999636 26067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:14.999733 26067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:40:15.005591 26067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:15.006780 26067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:15.007886 26067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:15.018939 26072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:15.019009 26073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:15.019184 26067 server_base.cc:1034] running on GCE node
W20260810 03:40:15.019013 26075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:15.019760 26067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:15.020498 26067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:15.021697 26067 hybrid_clock.cc:648] HybridClock initialized: now 1786333215021676 us; error 34 us; skew 500 ppm
Aug 10 03:40:15 dist-test-slave-ss4n krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333215, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:15.024813 26067 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:40:15.025888 26067 webserver.cc:460] Webserver started at http://127.21.175.65:42433/ using document root <none> and password file <none>
I20260810 03:40:15.026508 26067 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:15.026612 26067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:15.026914 26067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:15.029198 26067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "155bb39d16e94d9a865fc2aa8a4a6f98"
format_stamp: "Formatted at 2026-08-10 03:40:15 on dist-test-slave-ss4n"
server_key: "58a565e48828b7e944b9062d3b5a35f2"
server_key_iv: "cc9d5c44be3604ac6d8dc14cedef87b2"
server_key_version: "encryptionkey@0"
I20260810 03:40:15.029883 26067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "155bb39d16e94d9a865fc2aa8a4a6f98"
format_stamp: "Formatted at 2026-08-10 03:40:15 on dist-test-slave-ss4n"
server_key: "58a565e48828b7e944b9062d3b5a35f2"
server_key_iv: "cc9d5c44be3604ac6d8dc14cedef87b2"
server_key_version: "encryptionkey@0"
I20260810 03:40:15.034217 26067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 03:40:15.037269 26082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:15.038930 26067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:15.039122 26067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "155bb39d16e94d9a865fc2aa8a4a6f98"
format_stamp: "Formatted at 2026-08-10 03:40:15 on dist-test-slave-ss4n"
server_key: "58a565e48828b7e944b9062d3b5a35f2"
server_key_iv: "cc9d5c44be3604ac6d8dc14cedef87b2"
server_key_version: "encryptionkey@0"
I20260810 03:40:15.039278 26067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:15.083736 26067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:15.087492 26067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:15.087745 26067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:15.088390 26067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:15.089291 26067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:15.089362 26067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:15.089437 26067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:15.089536 26067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260810 03:40:15.099586 26067 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:32965
I20260810 03:40:15.099677 26202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:32965 every 8 connection(s)
I20260810 03:40:15.100714 26067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:40:15.111064 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 26067
I20260810 03:40:15.111189 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:40:15.111536 22205 external_mini_cluster.cc:1428] Setting key 728f4fcea2029dc36e932c0711701fd8
Aug 10 03:40:15 dist-test-slave-ss4n krb5kdc[25954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333215, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:15.116851 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:15.102921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:39661 (local address 127.21.175.126:35173)
0810 03:40:15.103096 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:15.103100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:15.104191 (+ 1091us) server_negotiation.cc:407] Connection header received
0810 03:40:15.105250 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:15.105255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:15.105321 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:15.105545 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:15.106860 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:15.107503 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:15.108815 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:15.109083 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:15.109213 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:15.112559 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:15.112563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:15.112565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:15.112594 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:15.114501 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:15.115096 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:15.115100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:15.115102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:15.115199 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:15.115591 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:15.115596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:15.115598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:15.115808 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:15.115938 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:15.116497 (+ 559us) server_negotiation.cc:299] Negotiation successful
0810 03:40:15.116663 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 03:40:15.117847 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:15.103231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:35173 (local address 127.21.175.65:39661)
0810 03:40:15.104005 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:15.104054 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:15.105010 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:15.105692 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:15.105701 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:15.106123 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:15.106658 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:15.106673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:15.107652 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:15.107661 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:15.108668 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:15.108678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:15.109261 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:15.109265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:15.109462 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:15.110265 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:15.110282 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:15.112319 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:15.114654 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:15.114661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:15.114679 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:15.114978 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:15.115303 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:15.115308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:15.115310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:15.115482 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:15.115927 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:15.115935 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:15.116225 (+ 290us) client_negotiation.cc:769] Sending connection context
0810 03:40:15.116446 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 03:40:15.116820 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":123,"threads_started":1}
I20260810 03:40:15.119297 26203 heartbeater.cc:347] Connected to a master server at 127.21.175.126:35173
I20260810 03:40:15.119593 26203 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:15.120157 26203 heartbeater.cc:511] Master 127.21.175.126:35173 requested a full tablet report, sending...
I20260810 03:40:15.122010 26002 ts_manager.cc:194] Registered new tserver with Master: 155bb39d16e94d9a865fc2aa8a4a6f98 (127.21.175.65:32965)
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
I20260810 03:40:15.123409 26002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:39661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:40:15.145574 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35173
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:15.267566 26208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:15.267900 26208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:15.267959 26208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:15.272742 26208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:15.272818 26208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:15.272888 26208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:40:15.279510 26208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:15.280711 26208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:15.281898 26208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:15.292806 26216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:15.292881 26208 server_base.cc:1034] running on GCE node
W20260810 03:40:15.292806 26213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:15.293176 26214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:15.293896 26208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:15.294647 26208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:15.295846 26208 hybrid_clock.cc:648] HybridClock initialized: now 1786333215295797 us; error 73 us; skew 500 ppm
Aug 10 03:40:15 dist-test-slave-ss4n krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333215, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:15.299288 26208 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:40:15.300539 26208 webserver.cc:460] Webserver started at http://127.21.175.66:42511/ using document root <none> and password file <none>
I20260810 03:40:15.301268 26208 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:15.301380 26208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:15.301729 26208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:15.304224 26208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "a9df0f14b18c4419a823091982ecdc9a"
format_stamp: "Formatted at 2026-08-10 03:40:15 on dist-test-slave-ss4n"
server_key: "bc5adf010d473aaaadfaa5ff8c50194c"
server_key_iv: "a9ee8c73383e8b24b226464df437d325"
server_key_version: "encryptionkey@0"
I20260810 03:40:15.304920 26208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "a9df0f14b18c4419a823091982ecdc9a"
format_stamp: "Formatted at 2026-08-10 03:40:15 on dist-test-slave-ss4n"
server_key: "bc5adf010d473aaaadfaa5ff8c50194c"
server_key_iv: "a9ee8c73383e8b24b226464df437d325"
server_key_version: "encryptionkey@0"
I20260810 03:40:15.309585 26208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260810 03:40:15.312892 26223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:15.314579 26208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:15.314736 26208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "a9df0f14b18c4419a823091982ecdc9a"
format_stamp: "Formatted at 2026-08-10 03:40:15 on dist-test-slave-ss4n"
server_key: "bc5adf010d473aaaadfaa5ff8c50194c"
server_key_iv: "a9ee8c73383e8b24b226464df437d325"
server_key_version: "encryptionkey@0"
I20260810 03:40:15.314873 26208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:15.331163 26208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:15.334950 26208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:15.335201 26208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:15.335919 26208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:15.336897 26208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:15.336971 26208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:15.337045 26208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:15.337096 26208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:15.347652 26208 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:43837
I20260810 03:40:15.347738 26342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:43837 every 8 connection(s)
I20260810 03:40:15.348744 26208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:40:15.353937 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 26208
I20260810 03:40:15.354074 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:40:15.354389 22205 external_mini_cluster.cc:1428] Setting key 9670f52b276d108087d08fd5a67a3366
Aug 10 03:40:15 dist-test-slave-ss4n krb5kdc[25954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333215, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:15.365835 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:15.350900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:41483 (local address 127.21.175.126:35173)
0810 03:40:15.351111 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:15.351117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:15.352163 (+ 1046us) server_negotiation.cc:407] Connection header received
0810 03:40:15.353721 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:15.353725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:15.353783 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:15.353867 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:15.355520 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:15.356171 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:15.358007 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:15.358364 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:15.358491 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:15.361746 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:15.361749 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:15.361752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:15.361786 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:15.363421 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:15.364028 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:15.364031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:15.364033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:15.364147 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:15.364528 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:15.364531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:15.364533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:15.364744 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:15.364860 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:15.365443 (+ 583us) server_negotiation.cc:299] Negotiation successful
0810 03:40:15.365639 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
I20260810 03:40:15.366768 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:15.351350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:35173 (local address 127.21.175.66:41483)
0810 03:40:15.351966 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:15.352015 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:15.353336 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:15.354137 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:15.354147 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:15.354678 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:15.355348 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:15.355368 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:15.356307 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:15.356310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:15.357862 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:15.357878 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:15.358535 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:15.358539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:15.358720 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:15.359438 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:15.359455 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:15.361453 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:15.363559 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:15.363566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:15.363586 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:15.363904 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:15.364249 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:15.364252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:15.364254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:15.364417 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:15.364861 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:15.364869 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:15.365177 (+ 308us) client_negotiation.cc:769] Sending connection context
0810 03:40:15.365384 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 03:40:15.365797 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":222,"threads_started":1}
I20260810 03:40:15.368199 26343 heartbeater.cc:347] Connected to a master server at 127.21.175.126:35173
I20260810 03:40:15.368531 26343 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:15.369143 26343 heartbeater.cc:511] Master 127.21.175.126:35173 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
I20260810 03:40:15.370625 26002 ts_manager.cc:194] Registered new tserver with Master: a9df0f14b18c4419a823091982ecdc9a (127.21.175.66:43837)
I20260810 03:40:15.371263 26002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:41483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:40:15.389217 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35173
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:15.509887 26348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:15.510192 26348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:15.510236 26348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:15.514777 26348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:15.514858 26348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:15.514923 26348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:40:15.520737 26348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:15.521938 26348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:15.522962 26348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:15.533727 26357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:15.533701 26354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:15.533710 26353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:15.533982 26348 server_base.cc:1034] running on GCE node
I20260810 03:40:15.534528 26348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:15.535271 26348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:15.536495 26348 hybrid_clock.cc:648] HybridClock initialized: now 1786333215536443 us; error 54 us; skew 500 ppm
Aug 10 03:40:15 dist-test-slave-ss4n krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333215, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:15.539851 26348 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:40:15.540879 26348 webserver.cc:460] Webserver started at http://127.21.175.67:44243/ using document root <none> and password file <none>
I20260810 03:40:15.541440 26348 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:15.541567 26348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:15.541836 26348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:15.544206 26348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "8b459544cd0d4e909c183f966bfe7842"
format_stamp: "Formatted at 2026-08-10 03:40:15 on dist-test-slave-ss4n"
server_key: "645823703e79511ef1a81b784d08e9aa"
server_key_iv: "5a1d911050e34150b2f6aaca605c688b"
server_key_version: "encryptionkey@0"
I20260810 03:40:15.544826 26348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "8b459544cd0d4e909c183f966bfe7842"
format_stamp: "Formatted at 2026-08-10 03:40:15 on dist-test-slave-ss4n"
server_key: "645823703e79511ef1a81b784d08e9aa"
server_key_iv: "5a1d911050e34150b2f6aaca605c688b"
server_key_version: "encryptionkey@0"
I20260810 03:40:15.549089 26348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260810 03:40:15.552636 26366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:15.554486 26348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:15.554666 26348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "8b459544cd0d4e909c183f966bfe7842"
format_stamp: "Formatted at 2026-08-10 03:40:15 on dist-test-slave-ss4n"
server_key: "645823703e79511ef1a81b784d08e9aa"
server_key_iv: "5a1d911050e34150b2f6aaca605c688b"
server_key_version: "encryptionkey@0"
I20260810 03:40:15.554817 26348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:15.586615 26348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:15.590649 26348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:15.590901 26348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:15.591614 26348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:15.592608 26348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:15.592691 26348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:15.592780 26348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:15.592829 26348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:15.603863 26348 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:40761
I20260810 03:40:15.603890 26486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:40761 every 8 connection(s)
I20260810 03:40:15.604918 26348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:40:15.608227 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 26348
I20260810 03:40:15.608388 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:40:15.608748 22205 external_mini_cluster.cc:1428] Setting key 4e72095a14537b34db8231526722c380
Aug 10 03:40:15 dist-test-slave-ss4n krb5kdc[25954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333215, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:15.622022 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:15.607228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:51957 (local address 127.21.175.126:35173)
0810 03:40:15.607372 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:15.607376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:15.608453 (+ 1077us) server_negotiation.cc:407] Connection header received
0810 03:40:15.610385 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:15.610389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:15.610449 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:15.610586 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:15.612196 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:15.612837 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:15.614354 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:15.614602 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:15.614723 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:15.617637 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:15.617641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:15.617643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:15.617671 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:15.619470 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:15.620237 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:15.620240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:15.620242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:15.620337 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:15.620732 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:15.620735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:15.620737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:15.620956 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:15.621058 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:15.621743 (+ 685us) server_negotiation.cc:299] Negotiation successful
0810 03:40:15.621868 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:40:15.623236 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:15.607620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:35173 (local address 127.21.175.67:51957)
0810 03:40:15.608249 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:15.608320 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:15.610121 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:15.610786 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:15.610797 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:15.611260 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:15.612004 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:15.612028 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:15.612989 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:15.612993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:15.614205 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:15.614219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:15.614779 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:15.614783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:15.614988 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:15.615650 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:15.615668 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:15.617350 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:15.619651 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:15.619659 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:15.619686 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:15.620128 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:15.620442 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:15.620444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:15.620446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:15.620614 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:15.621100 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:15.621106 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:15.621393 (+ 287us) client_negotiation.cc:769] Sending connection context
0810 03:40:15.621675 (+ 282us) client_negotiation.cc:240] Negotiation successful
0810 03:40:15.622042 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":133,"threads_started":1}
I20260810 03:40:15.624660 26487 heartbeater.cc:347] Connected to a master server at 127.21.175.126:35173
I20260810 03:40:15.624979 26487 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:15.625623 26487 heartbeater.cc:511] Master 127.21.175.126:35173 requested a full tablet report, sending...
I20260810 03:40:15.626824 26001 ts_manager.cc:194] Registered new tserver with Master: 8b459544cd0d4e909c183f966bfe7842 (127.21.175.67:40761)
I20260810 03:40:15.627483 26001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:51957
I20260810 03:40:15.636812 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:40:15 dist-test-slave-ss4n krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:40:15 dist-test-slave-ss4n krb5kdc[25954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.65@KRBTEST.COM
I20260810 03:40:15.667011 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:15.656717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.21.175.65:32965)
0810 03:40:15.657111 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:15.657122 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:15.657152 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:40:15.657219 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:15.657225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:15.657423 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:15.657629 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:15.658388 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:15.659317 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:15.660213 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:15.660752 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:15.660885 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:15.663604 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:15.663617 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:15.663635 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:15.663672 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:15.665213 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:15.665745 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:15.665750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:15.665752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:15.665837 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:15.666102 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:15.666104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:15.666106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:15.666365 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:15.666524 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:15.666736 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 03:40:15.666867 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":143,"threads_started":1}
Aug 10 03:40:15 dist-test-slave-ss4n krb5kdc[25954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:15.679065 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:15.670288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.21.175.126:35173)
0810 03:40:15.670450 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:15.670454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:15.670468 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:40:15.670604 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:15.670606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:15.670649 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:15.670718 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:15.671303 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:15.671871 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:15.672678 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:15.672937 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:15.673051 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:15.675882 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:15.675886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:15.675887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:15.675917 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:15.677410 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:15.677870 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:15.677873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:15.677874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:15.677954 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:15.678194 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:15.678196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:15.678197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:15.678429 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:15.678532 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:15.678746 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 03:40:15.678849 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 03:40:15.679507 26001 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56040
I20260810 03:40:15.680179 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 26067
I20260810 03:40:15.689066 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 26208
I20260810 03:40:15.697378 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 26348
I20260810 03:40:15.705338 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 25968
2026-08-10T03:40:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2488 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:40:15 dist-test-slave-ss4n krb5kdc[26499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:40:15 dist-test-slave-ss4n krb5kdc[26499](info): set up 2 sockets
Aug 10 03:40:15 dist-test-slave-ss4n krb5kdc[26499](info): commencing operation
krb5kdc: starting...
W20260810 03:40:16.872980 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:40:16 dist-test-slave-ss4n krb5kdc[26499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:40:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:40:17.018038 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:39539
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:39539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:17.138551 26513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:17.138845 26513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:17.138891 26513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:17.143008 26513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:17.143106 26513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:17.143128 26513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:17.143146 26513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:17.143162 26513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:40:17.148711 26513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:39539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:39539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:17.149889 26513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:17.150900 26513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:17.160014 26518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:17.160002 26519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:17.160002 26521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:17.160109 26513 server_base.cc:1034] running on GCE node
I20260810 03:40:17.160918 26513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:17.162211 26513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:17.163403 26513 hybrid_clock.cc:648] HybridClock initialized: now 1786333217163382 us; error 37 us; skew 500 ppm
Aug 10 03:40:17 dist-test-slave-ss4n krb5kdc[26499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333217, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:17.166882 26513 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:40:17.167752 26513 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:17.167825 26513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:17.168079 26513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:17.170542 26513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "d0c13b1208524680b14a793a149c632a"
format_stamp: "Formatted at 2026-08-10 03:40:17 on dist-test-slave-ss4n"
server_key: "f00cf8b62f6bcf4b8949c93db5e706a7"
server_key_iv: "bad9d4414917e60b70e9a5ab691dcf82"
server_key_version: "encryptionkey@0"
I20260810 03:40:17.171213 26513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "d0c13b1208524680b14a793a149c632a"
format_stamp: "Formatted at 2026-08-10 03:40:17 on dist-test-slave-ss4n"
server_key: "f00cf8b62f6bcf4b8949c93db5e706a7"
server_key_iv: "bad9d4414917e60b70e9a5ab691dcf82"
server_key_version: "encryptionkey@0"
I20260810 03:40:17.175468 26513 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:40:17.178318 26527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:17.179755 26513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:17.179901 26513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "d0c13b1208524680b14a793a149c632a"
format_stamp: "Formatted at 2026-08-10 03:40:17 on dist-test-slave-ss4n"
server_key: "f00cf8b62f6bcf4b8949c93db5e706a7"
server_key_iv: "bad9d4414917e60b70e9a5ab691dcf82"
server_key_version: "encryptionkey@0"
I20260810 03:40:17.180020 26513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:17.215559 26513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:17.219878 26513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:17.220140 26513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:17.228488 26513 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:39539
I20260810 03:40:17.228809 26580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:39539 every 8 connection(s)
I20260810 03:40:17.229337 26513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:17.232290 26581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:17.236683 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 26513
I20260810 03:40:17.236868 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:40:17.237180 22205 external_mini_cluster.cc:1428] Setting key da26d29c0541e561a363e3179fcd2c8d
I20260810 03:40:17.238479 26581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a: Bootstrap starting.
I20260810 03:40:17.240928 26581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:17.241832 26581 log.cc:824] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:17.243932 26581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a: No bootstrap required, opened a new log
Aug 10 03:40:17 dist-test-slave-ss4n krb5kdc[26499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:17.248279 26581 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c13b1208524680b14a793a149c632a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39539 } }
I20260810 03:40:17.248605 26581 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:17.248672 26581 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0c13b1208524680b14a793a149c632a, State: Initialized, Role: FOLLOWER
I20260810 03:40:17.249260 26581 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c13b1208524680b14a793a149c632a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39539 } }
I20260810 03:40:17.249447 26581 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:17.249542 26581 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:17.249656 26581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:17.250658 26581 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c13b1208524680b14a793a149c632a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39539 } }
I20260810 03:40:17.251034 26581 leader_election.cc:302] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0c13b1208524680b14a793a149c632a; no voters:
I20260810 03:40:17.251407 26581 leader_election.cc:288] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:40:17.251498 26587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:17.251730 26587 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [term 1 LEADER]: Becoming Leader. State: Replica: d0c13b1208524680b14a793a149c632a, State: Running, Role: LEADER
I20260810 03:40:17.252118 26587 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c13b1208524680b14a793a149c632a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39539 } }
I20260810 03:40:17.252879 26581 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:17.253397 26587 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0c13b1208524680b14a793a149c632a. Latest consensus state: current_term: 1 leader_uuid: "d0c13b1208524680b14a793a149c632a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c13b1208524680b14a793a149c632a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39539 } } }
I20260810 03:40:17.253619 26587 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:17.253922 26588 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0c13b1208524680b14a793a149c632a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c13b1208524680b14a793a149c632a" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39539 } } }
I20260810 03:40:17.254115 26593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:17.254114 26588 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:17.255064 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:17.238813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.21.175.126:39539)
0810 03:40:17.239294 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:17.239312 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:17.239353 (+ 41us) server_negotiation.cc:407] Connection header received
0810 03:40:17.240140 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:17.240160 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:17.240546 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:17.240940 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:17.241935 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:17.243103 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:17.244090 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:17.244655 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:17.244850 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:17.248048 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:17.248055 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:17.248075 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:17.248111 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:17.250577 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:17.251181 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:17.251191 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:17.251198 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:17.251350 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:17.251815 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:17.251819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:17.251821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:17.252409 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:17.252647 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:17.253436 (+ 789us) server_negotiation.cc:299] Negotiation successful
0810 03:40:17.253898 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":141,"threads_started":1}
I20260810 03:40:17.257685 26593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:17.264068 26593 catalog_manager.cc:1347] Generated new cluster ID: e5abc8163dc44fce8119ff5205b053f6
I20260810 03:40:17.264190 26593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:17.304416 26593 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:40:17.305706 26593 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:17.311518 26593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a: Generated new TSK 0
I20260810 03:40:17.312356 26593 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:40:17.349350 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39539
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:17.477367 26608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:17.477806 26608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:17.477895 26608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:17.482048 26608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:17.482146 26608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:17.482259 26608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:40:17.488019 26608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:17.489192 26608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:17.490301 26608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:17.500267 26615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:17.500267 26618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:17.500452 26616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:17.500808 26608 server_base.cc:1034] running on GCE node
I20260810 03:40:17.501338 26608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:17.502087 26608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:17.503283 26608 hybrid_clock.cc:648] HybridClock initialized: now 1786333217503233 us; error 64 us; skew 500 ppm
Aug 10 03:40:17 dist-test-slave-ss4n krb5kdc[26499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333217, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:17.506714 26608 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:40:17.507602 26608 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:17.507700 26608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:17.507951 26608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:17.510252 26608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "1447020574de4f18821999b45f886132"
format_stamp: "Formatted at 2026-08-10 03:40:17 on dist-test-slave-ss4n"
server_key: "5178375643018453e0bab64f6a6eeb4c"
server_key_iv: "0f69194f3c3f3ef5168539e264da00e4"
server_key_version: "encryptionkey@0"
I20260810 03:40:17.510859 26608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "1447020574de4f18821999b45f886132"
format_stamp: "Formatted at 2026-08-10 03:40:17 on dist-test-slave-ss4n"
server_key: "5178375643018453e0bab64f6a6eeb4c"
server_key_iv: "0f69194f3c3f3ef5168539e264da00e4"
server_key_version: "encryptionkey@0"
I20260810 03:40:17.515357 26608 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 03:40:17.518522 26624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:17.520006 26608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:17.520191 26608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "1447020574de4f18821999b45f886132"
format_stamp: "Formatted at 2026-08-10 03:40:17 on dist-test-slave-ss4n"
server_key: "5178375643018453e0bab64f6a6eeb4c"
server_key_iv: "0f69194f3c3f3ef5168539e264da00e4"
server_key_version: "encryptionkey@0"
I20260810 03:40:17.520336 26608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:17.537328 26608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:17.541378 26608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:17.541687 26608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:17.542476 26608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:17.543426 26608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:17.543514 26608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:17.543596 26608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:17.543654 26608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:17.554172 26608 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:44757
I20260810 03:40:17.554208 26741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:44757 every 8 connection(s)
I20260810 03:40:17.555002 26608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:40:17.558292 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 26608
I20260810 03:40:17.558449 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:40:17.558789 22205 external_mini_cluster.cc:1428] Setting key 7b521d7c692bae79ca909c654044c166
Aug 10 03:40:17 dist-test-slave-ss4n krb5kdc[26499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333217, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
I20260810 03:40:17.570969 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:17.557022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:43007 (local address 127.21.175.126:39539)
0810 03:40:17.557197 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:17.557201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:17.558319 (+ 1118us) server_negotiation.cc:407] Connection header received
0810 03:40:17.560019 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:17.560031 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:17.560091 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:17.560206 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:17.561965 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:17.562594 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:17.563849 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:17.564138 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:17.564231 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:17.567000 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:17.567006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:17.567013 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:17.567045 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:17.568606 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:17.569164 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:17.569168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:17.569170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:17.569264 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:17.569716 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:17.569719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:17.569720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:17.569953 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:17.570085 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:17.570639 (+ 554us) server_negotiation.cc:299] Negotiation successful
0810 03:40:17.570783 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:40:17.571852 26744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:17.557452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:39539 (local address 127.21.175.65:43007)
0810 03:40:17.558125 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:17.558190 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:17.559774 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:17.560406 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:17.560414 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:17.560972 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:17.561783 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:17.561807 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:17.562763 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:17.562767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:17.563708 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:17.563718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:17.564307 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:17.564310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:17.564463 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:17.565190 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:17.565208 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:17.566817 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:17.568747 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:17.568754 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:17.568772 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:17.569048 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:17.569364 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:17.569369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:17.569371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:17.569600 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:17.570085 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:17.570093 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:17.570376 (+ 283us) client_negotiation.cc:769] Sending connection context
0810 03:40:17.570584 (+ 208us) client_negotiation.cc:240] Negotiation successful
0810 03:40:17.570994 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
I20260810 03:40:17.573348 26742 heartbeater.cc:347] Connected to a master server at 127.21.175.126:39539
I20260810 03:40:17.573683 26742 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:17.574194 26742 heartbeater.cc:511] Master 127.21.175.126:39539 requested a full tablet report, sending...
I20260810 03:40:17.575933 26544 ts_manager.cc:194] Registered new tserver with Master: 1447020574de4f18821999b45f886132 (127.21.175.65:44757)
I20260810 03:40:17.577095 26544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:43007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:40:17.593127 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39539
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:17.712870 26748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:17.713189 26748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:17.713238 26748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:17.717765 26748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:17.717846 26748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:17.717938 26748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:40:17.723959 26748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:17.725236 26748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:17.726400 26748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:40:17.736809 26748 server_base.cc:1034] running on GCE node
W20260810 03:40:17.736760 26756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:17.736783 26754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:17.737119 26753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:17.737668 26748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:17.738377 26748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:17.739558 26748 hybrid_clock.cc:648] HybridClock initialized: now 1786333217739527 us; error 44 us; skew 500 ppm
Aug 10 03:40:17 dist-test-slave-ss4n krb5kdc[26499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333217, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:17.743032 26748 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:40:17.743858 26748 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:17.743952 26748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:17.744153 26748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:17.746574 26748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "1ce6d85eefea4a358f275db6a64f7b9c"
format_stamp: "Formatted at 2026-08-10 03:40:17 on dist-test-slave-ss4n"
server_key: "769eac05771519a6af8ae8aa35d5a36b"
server_key_iv: "0ab718fee83c4c59709bf7a16d4b5a72"
server_key_version: "encryptionkey@0"
I20260810 03:40:17.747179 26748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "1ce6d85eefea4a358f275db6a64f7b9c"
format_stamp: "Formatted at 2026-08-10 03:40:17 on dist-test-slave-ss4n"
server_key: "769eac05771519a6af8ae8aa35d5a36b"
server_key_iv: "0ab718fee83c4c59709bf7a16d4b5a72"
server_key_version: "encryptionkey@0"
I20260810 03:40:17.751567 26748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 03:40:17.754729 26762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:17.756194 26748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:40:17.756369 26748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "1ce6d85eefea4a358f275db6a64f7b9c"
format_stamp: "Formatted at 2026-08-10 03:40:17 on dist-test-slave-ss4n"
server_key: "769eac05771519a6af8ae8aa35d5a36b"
server_key_iv: "0ab718fee83c4c59709bf7a16d4b5a72"
server_key_version: "encryptionkey@0"
I20260810 03:40:17.756508 26748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:17.824226 26748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:17.827939 26748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:17.828187 26748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:17.828855 26748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:17.829893 26748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:17.829977 26748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:17.830082 26748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:17.830142 26748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:17.840173 26748 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:39143
I20260810 03:40:17.840191 26879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:39143 every 8 connection(s)
I20260810 03:40:17.840891 26748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:40:17.842481 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 26748
I20260810 03:40:17.842584 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:40:17.842972 22205 external_mini_cluster.cc:1428] Setting key 5cb4862f5d3f338c85a0c2801fff8941
Aug 10 03:40:17 dist-test-slave-ss4n krb5kdc[26499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333217, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
I20260810 03:40:17.857640 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:17.842862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:43649 (local address 127.21.175.126:39539)
0810 03:40:17.843050 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:17.843056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:17.844157 (+ 1101us) server_negotiation.cc:407] Connection header received
0810 03:40:17.846263 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:17.846266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:17.846313 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:17.846384 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:17.848030 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:17.848622 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:17.850060 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:17.850310 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:17.850436 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:17.853341 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:17.853345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:17.853348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:17.853380 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:17.855220 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:17.855810 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:17.855814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:17.855815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:17.855897 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:17.856315 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:17.856317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:17.856318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:17.856508 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:17.856586 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:17.857285 (+ 699us) server_negotiation.cc:299] Negotiation successful
0810 03:40:17.857439 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:40:17.858796 26882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:17.843317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:39539 (local address 127.21.175.66:43649)
0810 03:40:17.843950 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:17.844016 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:17.845996 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:17.846644 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:17.846658 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:17.847264 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:17.847844 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:17.847860 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:17.848774 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:17.848778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:17.849909 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:17.849921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:17.850499 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:17.850504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:17.850699 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:17.851430 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:17.851449 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:17.853142 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:17.855365 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:17.855371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:17.855385 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:17.855693 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:17.856007 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:17.856010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:17.856011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:17.856212 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:17.856606 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:17.856613 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:17.856997 (+ 384us) client_negotiation.cc:769] Sending connection context
0810 03:40:17.857217 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 03:40:17.857588 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":161,"threads_started":1}
I20260810 03:40:17.860220 26880 heartbeater.cc:347] Connected to a master server at 127.21.175.126:39539
I20260810 03:40:17.860498 26880 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:17.861016 26880 heartbeater.cc:511] Master 127.21.175.126:39539 requested a full tablet report, sending...
I20260810 03:40:17.862249 26544 ts_manager.cc:194] Registered new tserver with Master: 1ce6d85eefea4a358f275db6a64f7b9c (127.21.175.66:39143)
I20260810 03:40:17.862900 26544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:43649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:40:17.877019 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39539
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:17.998253 26885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:17.998545 26885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:17.998592 26885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:18.002707 26885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:18.002779 26885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:18.002841 26885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:40:18.008324 26885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:18.009516 26885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:18.010599 26885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:18.021024 26891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:18.021034 26890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:18.021034 26893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:18.021095 26885 server_base.cc:1034] running on GCE node
I20260810 03:40:18.021876 26885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:18.022565 26885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:18.023738 26885 hybrid_clock.cc:648] HybridClock initialized: now 1786333218023710 us; error 36 us; skew 500 ppm
Aug 10 03:40:18 dist-test-slave-ss4n krb5kdc[26499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333218, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:18.026978 26885 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:40:18.027809 26885 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:18.027876 26885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:18.028048 26885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:18.030220 26885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "c838920839334973a2876ab618947ac8"
format_stamp: "Formatted at 2026-08-10 03:40:18 on dist-test-slave-ss4n"
server_key: "5168f406846485c25e7524bb4dca361f"
server_key_iv: "1a9f098b21f5fe6819513817240f128d"
server_key_version: "encryptionkey@0"
I20260810 03:40:18.030772 26885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "c838920839334973a2876ab618947ac8"
format_stamp: "Formatted at 2026-08-10 03:40:18 on dist-test-slave-ss4n"
server_key: "5168f406846485c25e7524bb4dca361f"
server_key_iv: "1a9f098b21f5fe6819513817240f128d"
server_key_version: "encryptionkey@0"
I20260810 03:40:18.034838 26885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:40:18.037720 26899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:18.039222 26885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:18.039379 26885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "c838920839334973a2876ab618947ac8"
format_stamp: "Formatted at 2026-08-10 03:40:18 on dist-test-slave-ss4n"
server_key: "5168f406846485c25e7524bb4dca361f"
server_key_iv: "1a9f098b21f5fe6819513817240f128d"
server_key_version: "encryptionkey@0"
I20260810 03:40:18.039510 26885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:18.056099 26885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:18.059778 26885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:18.060001 26885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:18.060684 26885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:18.061594 26885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:18.061668 26885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:18.061734 26885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:18.061807 26885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:18.071267 26885 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:43077
I20260810 03:40:18.071343 27014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:43077 every 8 connection(s)
I20260810 03:40:18.072125 26885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:40:18.075215 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 26885
I20260810 03:40:18.075361 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:40:18.075722 22205 external_mini_cluster.cc:1428] Setting key 7b42de2cae4eafe8745f0e9167e01c35
Aug 10 03:40:18 dist-test-slave-ss4n krb5kdc[26499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333218, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:18.087318 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.074637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:60167 (local address 127.21.175.126:39539)
0810 03:40:18.074805 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:18.074808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:18.075867 (+ 1059us) server_negotiation.cc:407] Connection header received
0810 03:40:18.077353 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:18.077355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:18.077401 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:18.077475 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:18.078782 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.079327 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.080486 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.080734 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.080832 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:18.083408 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:18.083411 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:18.083414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:18.083441 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:18.085105 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:18.085631 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:18.085634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:18.085635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:18.085730 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:18.086060 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:18.086063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:18.086064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:18.086265 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:18.086349 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:18.087054 (+ 705us) server_negotiation.cc:299] Negotiation successful
0810 03:40:18.087184 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 03:40:18.088197 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.075118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:39539 (local address 127.21.175.67:60167)
0810 03:40:18.075689 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:18.075736 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:18.077094 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:18.077693 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:18.077702 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:18.078135 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:18.078639 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:18.078654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.079439 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.079441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:18.080324 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:18.080332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.080876 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.080879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:18.081022 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:18.081705 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:18.081722 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:18.083220 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:18.085214 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:18.085219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:18.085232 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:18.085544 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:18.085815 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:18.085817 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:18.085818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:18.085961 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:18.086421 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:18.086427 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:18.086760 (+ 333us) client_negotiation.cc:769] Sending connection context
0810 03:40:18.086950 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 03:40:18.087250 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":160,"threads_started":1}
I20260810 03:40:18.089773 27015 heartbeater.cc:347] Connected to a master server at 127.21.175.126:39539
I20260810 03:40:18.090082 27015 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:18.090669 27015 heartbeater.cc:511] Master 127.21.175.126:39539 requested a full tablet report, sending...
I20260810 03:40:18.091912 26543 ts_manager.cc:194] Registered new tserver with Master: c838920839334973a2876ab618947ac8 (127.21.175.67:43077)
I20260810 03:40:18.092564 26543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:60167
I20260810 03:40:18.102952 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:40:18.114179 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.105213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33482 (local address 127.21.175.126:39539)
0810 03:40:18.105435 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:18.105442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:18.105713 (+ 271us) server_negotiation.cc:407] Connection header received
0810 03:40:18.105888 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:18.105899 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:18.105953 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:18.106054 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:18.106868 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.107473 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.108429 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.108682 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.108804 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:18.110458 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:18.110464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:18.110472 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:18.110500 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:18.112095 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:18.112603 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:18.112607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:18.112609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:18.112701 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:18.113051 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:18.113055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:18.113057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:18.113315 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:18.113479 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:18.113844 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 03:40:18.113994 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:40:18.117652 26543 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:40:18.120922 26543 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:40:18.137259 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.131532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:39143 (local address 127.0.0.1:39686)
0810 03:40:18.132126 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:18.132160 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:18.132262 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:18.132732 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:18.132735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:18.132754 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:18.132871 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:18.132882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.134446 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.134450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:18.135766 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:18.135778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.136868 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.136871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:18.136978 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:18.137000 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 03:40:18.137065 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 03:40:18.137129 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":97,"thread_start_us":89,"threads_started":1}
I20260810 03:40:18.137591 27031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.131493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43068 (local address 127.21.175.67:43077)
0810 03:40:18.131901 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:18.131913 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:18.131940 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:40:18.132035 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:18.132040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:18.132287 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:18.132445 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:18.132800 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.133733 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.135481 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.136467 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.136589 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:18.136754 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:18.137091 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 03:40:18.137205 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":139,"threads_started":1}
I20260810 03:40:18.137689 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.130910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.21.175.65:44757)
0810 03:40:18.131356 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:18.131365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:18.132197 (+ 832us) server_negotiation.cc:407] Connection header received
0810 03:40:18.132288 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:18.132295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:18.132517 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:18.132670 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:18.133035 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.134298 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.135926 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.136744 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.136837 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:18.136979 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:18.137368 (+ 389us) server_negotiation.cc:299] Negotiation successful
0810 03:40:18.137531 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":98,"threads_started":1}
I20260810 03:40:18.137547 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.130721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:44757 (local address 127.0.0.1:34332)
0810 03:40:18.132062 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:18.132076 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:18.132188 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:18.132817 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:18.132820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:18.132838 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:18.132936 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:18.132941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.134446 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.134450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:18.135770 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:18.135778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.137085 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.137088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:18.137190 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:18.137212 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 03:40:18.137319 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 03:40:18.137389 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1240,"mutex_wait_us":53,"thread_start_us":117,"threads_started":1}
I20260810 03:40:18.138219 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.131209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39686 (local address 127.21.175.66:39143)
0810 03:40:18.131886 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:18.131898 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:18.132298 (+ 400us) server_negotiation.cc:407] Connection header received
0810 03:40:18.132387 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:18.132394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:18.132579 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:18.132753 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:18.133135 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.134295 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.135916 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.136744 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.136854 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:18.136991 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:18.137833 (+ 842us) server_negotiation.cc:299] Negotiation successful
0810 03:40:18.137954 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":119,"threads_started":1}
I20260810 03:40:18.137259 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.130607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:43077 (local address 127.0.0.1:43068)
0810 03:40:18.131615 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:18.131671 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:18.131794 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:18.132444 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:18.132450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:18.132471 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:18.132659 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:18.132668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.133862 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.133869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:18.135353 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:18.135363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.136629 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.136633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:18.136751 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:18.136831 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 03:40:18.136945 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 03:40:18.137087 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":91,"thread_start_us":128,"threads_started":1}
I20260810 03:40:18.140918 26675 tablet_service.cc:1467] Processing CreateTablet for tablet 70a80e19afed4e3785076803948570b7 (DEFAULT_TABLE table=test-table [id=1671ee1be26e4353aa5ed642313415ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:18.140913 26811 tablet_service.cc:1467] Processing CreateTablet for tablet 70a80e19afed4e3785076803948570b7 (DEFAULT_TABLE table=test-table [id=1671ee1be26e4353aa5ed642313415ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:18.142042 26811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70a80e19afed4e3785076803948570b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:18.144207 26949 tablet_service.cc:1467] Processing CreateTablet for tablet 70a80e19afed4e3785076803948570b7 (DEFAULT_TABLE table=test-table [id=1671ee1be26e4353aa5ed642313415ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:18.142053 26675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70a80e19afed4e3785076803948570b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:18.145288 26949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70a80e19afed4e3785076803948570b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:18.149554 27032 tablet_bootstrap.cc:492] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c: Bootstrap starting.
I20260810 03:40:18.151847 27033 tablet_bootstrap.cc:492] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132: Bootstrap starting.
I20260810 03:40:18.152940 27034 tablet_bootstrap.cc:492] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8: Bootstrap starting.
I20260810 03:40:18.152966 27032 tablet_bootstrap.cc:654] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:18.154489 27033 tablet_bootstrap.cc:654] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:18.154762 27032 log.cc:824] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:18.155100 27034 tablet_bootstrap.cc:654] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:18.155306 27033 log.cc:824] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:18.156322 27034 log.cc:824] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:18.157289 27032 tablet_bootstrap.cc:492] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c: No bootstrap required, opened a new log
I20260810 03:40:18.157370 27033 tablet_bootstrap.cc:492] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132: No bootstrap required, opened a new log
I20260810 03:40:18.157553 27032 ts_tablet_manager.cc:1397] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260810 03:40:18.157580 27033 ts_tablet_manager.cc:1397] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:40:18.158356 27034 tablet_bootstrap.cc:492] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8: No bootstrap required, opened a new log
I20260810 03:40:18.158547 27034 ts_tablet_manager.cc:1397] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:40:18.161697 27032 raft_consensus.cc:348] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c838920839334973a2876ab618947ac8" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 43077 } } peers { permanent_uuid: "1447020574de4f18821999b45f886132" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44757 } } peers { permanent_uuid: "1ce6d85eefea4a358f275db6a64f7b9c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39143 } }
I20260810 03:40:18.162002 27032 raft_consensus.cc:374] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:18.162070 27032 raft_consensus.cc:729] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ce6d85eefea4a358f275db6a64f7b9c, State: Initialized, Role: FOLLOWER
I20260810 03:40:18.162284 27033 raft_consensus.cc:348] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c838920839334973a2876ab618947ac8" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 43077 } } peers { permanent_uuid: "1447020574de4f18821999b45f886132" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44757 } } peers { permanent_uuid: "1ce6d85eefea4a358f275db6a64f7b9c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39143 } }
I20260810 03:40:18.162549 27033 raft_consensus.cc:374] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:18.162406 27034 raft_consensus.cc:348] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c838920839334973a2876ab618947ac8" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 43077 } } peers { permanent_uuid: "1447020574de4f18821999b45f886132" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44757 } } peers { permanent_uuid: "1ce6d85eefea4a358f275db6a64f7b9c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39143 } }
I20260810 03:40:18.162626 27034 raft_consensus.cc:374] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:18.162631 27033 raft_consensus.cc:729] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1447020574de4f18821999b45f886132, State: Initialized, Role: FOLLOWER
I20260810 03:40:18.162683 27034 raft_consensus.cc:729] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c838920839334973a2876ab618947ac8, State: Initialized, Role: FOLLOWER
I20260810 03:40:18.162633 27032 consensus_queue.cc:260] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c838920839334973a2876ab618947ac8" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 43077 } } peers { permanent_uuid: "1447020574de4f18821999b45f886132" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44757 } } peers { permanent_uuid: "1ce6d85eefea4a358f275db6a64f7b9c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39143 } }
I20260810 03:40:18.163153 27034 consensus_queue.cc:260] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c838920839334973a2876ab618947ac8" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 43077 } } peers { permanent_uuid: "1447020574de4f18821999b45f886132" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44757 } } peers { permanent_uuid: "1ce6d85eefea4a358f275db6a64f7b9c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39143 } }
I20260810 03:40:18.163137 27033 consensus_queue.cc:260] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c838920839334973a2876ab618947ac8" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 43077 } } peers { permanent_uuid: "1447020574de4f18821999b45f886132" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44757 } } peers { permanent_uuid: "1ce6d85eefea4a358f275db6a64f7b9c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39143 } }
I20260810 03:40:18.163582 26880 heartbeater.cc:503] Master 127.21.175.126:39539 was elected leader, sending a full tablet report...
I20260810 03:40:18.163849 27032 ts_tablet_manager.cc:1428] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:40:18.164100 27034 ts_tablet_manager.cc:1428] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 03:40:18.164500 27015 heartbeater.cc:503] Master 127.21.175.126:39539 was elected leader, sending a full tablet report...
I20260810 03:40:18.164928 26742 heartbeater.cc:503] Master 127.21.175.126:39539 was elected leader, sending a full tablet report...
I20260810 03:40:18.165248 27033 ts_tablet_manager.cc:1428] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 03:40:18.276010 27040 raft_consensus.cc:482] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:18.276262 27040 raft_consensus.cc:504] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c838920839334973a2876ab618947ac8" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 43077 } } peers { permanent_uuid: "1447020574de4f18821999b45f886132" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44757 } } peers { permanent_uuid: "1ce6d85eefea4a358f275db6a64f7b9c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39143 } }
I20260810 03:40:18.277647 27040 leader_election.cc:288] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1447020574de4f18821999b45f886132 (127.21.175.65:44757), 1ce6d85eefea4a358f275db6a64f7b9c (127.21.175.66:39143)
I20260810 03:40:18.281703 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.278039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:58399 (local address 127.21.175.66:39143)
0810 03:40:18.278184 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:18.278188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:18.278489 (+ 301us) server_negotiation.cc:407] Connection header received
0810 03:40:18.278664 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:18.278667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:18.278710 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:18.278797 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:18.279065 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.279586 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.280679 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.281100 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.281233 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:18.281290 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:18.281472 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 03:40:18.281575 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:40:18.281859 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.277894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:44757 (local address 127.21.175.67:60851)
0810 03:40:18.278124 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:18.278141 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:18.278256 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:18.278575 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:18.278578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:18.278596 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:18.278728 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:18.278734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.279566 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.279571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:18.280539 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:18.280549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.281300 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.281304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:18.281449 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:18.281467 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 03:40:18.281618 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 03:40:18.281698 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":59}
I20260810 03:40:18.281857 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.278026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:60851 (local address 127.21.175.65:44757)
0810 03:40:18.278169 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:18.278177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:18.278196 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:40:18.278403 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:18.278412 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:18.278460 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:18.278560 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:18.278855 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.279423 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.280688 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.281163 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.281309 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:18.281373 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:18.281637 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 03:40:18.281719 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:40:18.281724 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.277955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:39143 (local address 127.21.175.67:58399)
0810 03:40:18.278377 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:18.278389 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:18.278537 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:18.278806 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:18.278810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:18.278824 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:18.278947 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:18.278953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.279707 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.279711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:18.280546 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:18.280555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.281232 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.281236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:18.281337 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:18.281361 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:40:18.281454 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 03:40:18.281556 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":2,"thread_start_us":140,"threads_started":1}
I20260810 03:40:18.282689 26695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a80e19afed4e3785076803948570b7" candidate_uuid: "c838920839334973a2876ab618947ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1447020574de4f18821999b45f886132" is_pre_election: true
I20260810 03:40:18.282825 26832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a80e19afed4e3785076803948570b7" candidate_uuid: "c838920839334973a2876ab618947ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ce6d85eefea4a358f275db6a64f7b9c" is_pre_election: true
I20260810 03:40:18.283125 26832 raft_consensus.cc:2454] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c838920839334973a2876ab618947ac8 in term 0.
I20260810 03:40:18.283126 26695 raft_consensus.cc:2454] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c838920839334973a2876ab618947ac8 in term 0.
I20260810 03:40:18.283732 26902 leader_election.cc:302] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1447020574de4f18821999b45f886132, 1ce6d85eefea4a358f275db6a64f7b9c, c838920839334973a2876ab618947ac8; no voters:
I20260810 03:40:18.284119 27040 raft_consensus.cc:2781] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:40:18.284224 27040 raft_consensus.cc:482] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:40:18.284272 27040 raft_consensus.cc:3037] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:18.285470 27040 raft_consensus.cc:504] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c838920839334973a2876ab618947ac8" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 43077 } } peers { permanent_uuid: "1447020574de4f18821999b45f886132" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44757 } } peers { permanent_uuid: "1ce6d85eefea4a358f275db6a64f7b9c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39143 } }
I20260810 03:40:18.285979 27040 leader_election.cc:288] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [CANDIDATE]: Term 1 election: Requested vote from peers 1447020574de4f18821999b45f886132 (127.21.175.65:44757), 1ce6d85eefea4a358f275db6a64f7b9c (127.21.175.66:39143)
I20260810 03:40:18.286412 26695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a80e19afed4e3785076803948570b7" candidate_uuid: "c838920839334973a2876ab618947ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1447020574de4f18821999b45f886132"
I20260810 03:40:18.286597 26695 raft_consensus.cc:3037] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:18.286616 26832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a80e19afed4e3785076803948570b7" candidate_uuid: "c838920839334973a2876ab618947ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ce6d85eefea4a358f275db6a64f7b9c"
I20260810 03:40:18.286777 26832 raft_consensus.cc:3037] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:18.287840 26695 raft_consensus.cc:2454] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c838920839334973a2876ab618947ac8 in term 1.
I20260810 03:40:18.287914 26832 raft_consensus.cc:2454] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c838920839334973a2876ab618947ac8 in term 1.
I20260810 03:40:18.288230 26902 leader_election.cc:302] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1447020574de4f18821999b45f886132, c838920839334973a2876ab618947ac8; no voters:
I20260810 03:40:18.288494 27040 raft_consensus.cc:2781] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:18.288749 27040 raft_consensus.cc:686] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [term 1 LEADER]: Becoming Leader. State: Replica: c838920839334973a2876ab618947ac8, State: Running, Role: LEADER
I20260810 03:40:18.289165 27040 consensus_queue.cc:237] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c838920839334973a2876ab618947ac8" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 43077 } } peers { permanent_uuid: "1447020574de4f18821999b45f886132" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44757 } } peers { permanent_uuid: "1ce6d85eefea4a358f275db6a64f7b9c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39143 } }
I20260810 03:40:18.292845 26544 catalog_manager.cc:5495] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c838920839334973a2876ab618947ac8 (127.21.175.67). New cstate: current_term: 1 leader_uuid: "c838920839334973a2876ab618947ac8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c838920839334973a2876ab618947ac8" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 43077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1447020574de4f18821999b45f886132" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ce6d85eefea4a358f275db6a64f7b9c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39143 } health_report { overall_health: UNKNOWN } } }
W20260810 03:40:18.306497 26743 tablet.cc:2369] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:40:18.323372 27016 tablet.cc:2369] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:40:18.330658 27031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.326578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43084 (local address 127.21.175.67:43077)
0810 03:40:18.326711 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:18.326715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:18.326747 (+ 32us) server_negotiation.cc:407] Connection header received
0810 03:40:18.326823 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:18.326825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:18.326882 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:18.326966 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:40:18.327246 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.327839 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.328907 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:18.329189 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:18.329285 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:18.329678 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:40:18.330169 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:40:18.330269 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:18.330396 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 03:40:18.330456 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:40:18.337237 26695 raft_consensus.cc:1261] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132 [term 1 FOLLOWER]: Refusing update from remote peer c838920839334973a2876ab618947ac8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:18.337237 26832 raft_consensus.cc:1261] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c [term 1 FOLLOWER]: Refusing update from remote peer c838920839334973a2876ab618947ac8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:18.338115 27043 consensus_queue.cc:1035] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ce6d85eefea4a358f275db6a64f7b9c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:18.338374 27040 consensus_queue.cc:1035] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1447020574de4f18821999b45f886132" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 03:40:18.342024 26881 tablet.cc:2369] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:40:18.348003 27047 mvcc.cc:205] Tried to move back new op lower bound from 7316820862301315072 to 7316820862115926016. Current Snapshot: MvccSnapshot[applied={T|T < 7316820862301315072}]
I20260810 03:40:18.348300 27046 mvcc.cc:205] Tried to move back new op lower bound from 7316820862301315072 to 7316820862115926016. Current Snapshot: MvccSnapshot[applied={T|T < 7316820862301315072}]
I20260810 03:40:18.358642 26544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:18.358867 26544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:18.362174 26544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0c13b1208524680b14a793a149c632a: Sending DeleteTablet for 3 replicas of tablet 70a80e19afed4e3785076803948570b7
I20260810 03:40:18.363169 26949 tablet_service.cc:1514] Processing DeleteTablet for tablet 70a80e19afed4e3785076803948570b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:18 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:43068
I20260810 03:40:18.363220 26675 tablet_service.cc:1514] Processing DeleteTablet for tablet 70a80e19afed4e3785076803948570b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:18 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:34332
I20260810 03:40:18.363816 27057 tablet_replica.cc:330] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8: stopping tablet replica
I20260810 03:40:18.363816 27056 tablet_replica.cc:330] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132: stopping tablet replica
I20260810 03:40:18.363782 26811 tablet_service.cc:1514] Processing DeleteTablet for tablet 70a80e19afed4e3785076803948570b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:18 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:39686
I20260810 03:40:18.364233 27056 raft_consensus.cc:2229] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:18.364233 27057 raft_consensus.cc:2229] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:40:18.364701 27056 raft_consensus.cc:2258] T 70a80e19afed4e3785076803948570b7 P 1447020574de4f18821999b45f886132 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:18.364696 27057 raft_consensus.cc:2258] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:18.365116 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 26608
I20260810 03:40:18.367393 27057 ts_tablet_manager.cc:1905] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:18.370198 27057 ts_tablet_manager.cc:1918] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:40:18.370337 27057 log.cc:1196] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/70a80e19afed4e3785076803948570b7
I20260810 03:40:18.370679 27057 ts_tablet_manager.cc:1939] T 70a80e19afed4e3785076803948570b7 P c838920839334973a2876ab618947ac8: Deleting consensus metadata
I20260810 03:40:18.372001 26529 catalog_manager.cc:4841] TS c838920839334973a2876ab618947ac8 (127.21.175.67:43077): tablet 70a80e19afed4e3785076803948570b7 (table test-table [id=1671ee1be26e4353aa5ed642313415ff]) successfully deleted
I20260810 03:40:18.373795 27058 tablet_replica.cc:330] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c: stopping tablet replica
I20260810 03:40:18.374150 27058 raft_consensus.cc:2229] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:18.374500 27058 raft_consensus.cc:2258] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:18.375914 27058 ts_tablet_manager.cc:1905] T 70a80e19afed4e3785076803948570b7 P 1ce6d85eefea4a358f275db6a64f7b9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 03:40:18.376952 26530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.175.65:44757 (error 108)
I20260810 03:40:18.378130 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.377802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:44757 (local address 127.0.0.1:34346)
0810 03:40:18.377957 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:18.378032 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:44757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 03:40:18.378197 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 26748
W20260810 03:40:18.382256 26530 catalog_manager.cc:4584] TS 1447020574de4f18821999b45f886132 (127.21.175.65:44757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70a80e19afed4e3785076803948570b7: Network error: Client connection negotiation failed: client connection to 127.21.175.65:44757: connect: Connection refused (error 111)
I20260810 03:40:18.387951 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 26885
I20260810 03:40:18.388192 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.387843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:39143 (local address 127.0.0.1:39696)
0810 03:40:18.388012 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:18.388083 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.66:39143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 03:40:18.395587 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 26513
2026-08-10T03:40:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2689 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T03:40:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:18Z Disabled control of system clock
I20260810 03:40:18.431927 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:44325
--webserver_interface=127.21.175.126
--webserver_port=0
--builtin_ntp_servers=127.21.175.84:34635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:44325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 03:40:18.558657 27063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:18.558945 27063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:18.558983 27063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:18.563036 27063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:18.563107 27063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:18.563122 27063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:18.563138 27063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:18.563154 27063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:40:18.567993 27063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:34635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:44325
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:44325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:18.569247 27063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:18.570650 27063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:18.580121 27069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:18.580137 27063 server_base.cc:1034] running on GCE node
W20260810 03:40:18.580170 27071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:18.580159 27068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:18.580942 27063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:18.582166 27063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:18.583338 27063 hybrid_clock.cc:648] HybridClock initialized: now 1786333218583325 us; error 28 us; skew 500 ppm
I20260810 03:40:18.585379 27063 webserver.cc:460] Webserver started at http://127.21.175.126:41969/ using document root <none> and password file <none>
I20260810 03:40:18.586014 27063 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:18.586082 27063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:18.586319 27063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:18.588456 27063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "dd7a07b10bfa4a08a37ac916db69bac4"
format_stamp: "Formatted at 2026-08-10 03:40:18 on dist-test-slave-ss4n"
server_key: "2a590db33086d422a388f3f77693adc5"
server_key_iv: "79f5819ef873b4f664f9eefcbf5ffe52"
server_key_version: "encryptionkey@0"
I20260810 03:40:18.589061 27063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "dd7a07b10bfa4a08a37ac916db69bac4"
format_stamp: "Formatted at 2026-08-10 03:40:18 on dist-test-slave-ss4n"
server_key: "2a590db33086d422a388f3f77693adc5"
server_key_iv: "79f5819ef873b4f664f9eefcbf5ffe52"
server_key_version: "encryptionkey@0"
I20260810 03:40:18.593015 27063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 03:40:18.595764 27078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:18.597179 27063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:18.597429 27063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "dd7a07b10bfa4a08a37ac916db69bac4"
format_stamp: "Formatted at 2026-08-10 03:40:18 on dist-test-slave-ss4n"
server_key: "2a590db33086d422a388f3f77693adc5"
server_key_iv: "79f5819ef873b4f664f9eefcbf5ffe52"
server_key_version: "encryptionkey@0"
I20260810 03:40:18.597663 27063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:18.603173 27063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:18.603886 27063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:18.604115 27063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:18.612358 27063 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:44325
I20260810 03:40:18.612376 27132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:44325 every 8 connection(s)
I20260810 03:40:18.613456 27063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:18.618127 27133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:18.619282 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 27063
I20260810 03:40:18.619467 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:40:18.619884 22205 external_mini_cluster.cc:1428] Setting key 007327991aacfe0889a2d9dd5cb987ef
I20260810 03:40:18.625890 27133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4: Bootstrap starting.
I20260810 03:40:18.626698 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.621594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.21.175.126:44325)
0810 03:40:18.622407 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:18.622429 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:18.622476 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:40:18.623357 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:18.623383 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:18.623867 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:18.624306 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:18.625148 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:18.625154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:18.625169 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:18.625181 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:18.625251 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:18.625275 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:18.625425 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:18.625525 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 03:40:18.625780 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":146,"threads_started":1}
I20260810 03:40:18.628582 27133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:18.629351 27133 log.cc:824] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:18.631397 27133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4: No bootstrap required, opened a new log
I20260810 03:40:18.635028 27133 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7a07b10bfa4a08a37ac916db69bac4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44325 } }
I20260810 03:40:18.635285 27133 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:18.635352 27133 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7a07b10bfa4a08a37ac916db69bac4, State: Initialized, Role: FOLLOWER
I20260810 03:40:18.635788 27133 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7a07b10bfa4a08a37ac916db69bac4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44325 } }
I20260810 03:40:18.635934 27133 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:18.636001 27133 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:18.636129 27133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:18.637178 27133 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7a07b10bfa4a08a37ac916db69bac4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44325 } }
I20260810 03:40:18.637609 27133 leader_election.cc:302] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd7a07b10bfa4a08a37ac916db69bac4; no voters:
I20260810 03:40:18.637949 27133 leader_election.cc:288] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:40:18.638134 27138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:18.638439 27138 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [term 1 LEADER]: Becoming Leader. State: Replica: dd7a07b10bfa4a08a37ac916db69bac4, State: Running, Role: LEADER
I20260810 03:40:18.638824 27138 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7a07b10bfa4a08a37ac916db69bac4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44325 } }
I20260810 03:40:18.639305 27133 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:18.640653 27141 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd7a07b10bfa4a08a37ac916db69bac4. Latest consensus state: current_term: 1 leader_uuid: "dd7a07b10bfa4a08a37ac916db69bac4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7a07b10bfa4a08a37ac916db69bac4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44325 } } }
I20260810 03:40:18.640799 27141 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:18.641149 27139 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd7a07b10bfa4a08a37ac916db69bac4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7a07b10bfa4a08a37ac916db69bac4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44325 } } }
I20260810 03:40:18.641253 27139 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4 [sys.catalog]: This master's current role is: LEADER
W20260810 03:40:18.644343 27154 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:40:18.644448 27154 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:40:18.644526 27148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:18.645236 27148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:18.651247 27148 catalog_manager.cc:1347] Generated new cluster ID: 1bffc7ce202942b495087937ce81f05e
I20260810 03:40:18.651332 27148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:18.672271 27148 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:40:18.672703 27148 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:18.685271 27148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4: Generated new TSK 0
I20260810 03:40:18.686316 27148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:40:18.695199 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:44325
--builtin_ntp_servers=127.21.175.84:34635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 03:40:18.822310 27159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:18.822678 27159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:18.822772 27159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:18.827462 27159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:18.827592 27159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:18.827705 27159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:40:18.833256 27159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:34635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:44325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:18.834496 27159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:18.836024 27159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:18.846354 27164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:18.846406 27159 server_base.cc:1034] running on GCE node
W20260810 03:40:18.846357 27165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:18.846356 27167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:18.846976 27159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:18.847707 27159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:18.848906 27159 hybrid_clock.cc:648] HybridClock initialized: now 1786333218848887 us; error 30 us; skew 500 ppm
I20260810 03:40:18.850895 27159 webserver.cc:460] Webserver started at http://127.21.175.65:42041/ using document root <none> and password file <none>
I20260810 03:40:18.851516 27159 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:18.851620 27159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:18.851864 27159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:18.854076 27159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "e485a606249a4a9188ab6ca47da8af06"
format_stamp: "Formatted at 2026-08-10 03:40:18 on dist-test-slave-ss4n"
server_key: "4333e9df249972d22744727383e27329"
server_key_iv: "0c4017b9260fd2821e86b6be6c677c26"
server_key_version: "encryptionkey@0"
I20260810 03:40:18.854681 27159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "e485a606249a4a9188ab6ca47da8af06"
format_stamp: "Formatted at 2026-08-10 03:40:18 on dist-test-slave-ss4n"
server_key: "4333e9df249972d22744727383e27329"
server_key_iv: "0c4017b9260fd2821e86b6be6c677c26"
server_key_version: "encryptionkey@0"
I20260810 03:40:18.858589 27159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:40:18.861334 27173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:18.862761 27159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:40:18.862913 27159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "e485a606249a4a9188ab6ca47da8af06"
format_stamp: "Formatted at 2026-08-10 03:40:18 on dist-test-slave-ss4n"
server_key: "4333e9df249972d22744727383e27329"
server_key_iv: "0c4017b9260fd2821e86b6be6c677c26"
server_key_version: "encryptionkey@0"
I20260810 03:40:18.863042 27159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:18.867853 27159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:18.868494 27159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:18.868690 27159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:18.869251 27159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:18.870251 27159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:18.870321 27159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:18.870397 27159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:18.870447 27159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:18.879231 27159 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:34811
I20260810 03:40:18.879302 27293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:34811 every 8 connection(s)
I20260810 03:40:18.880250 27159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:40:18.883325 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 27159
I20260810 03:40:18.883455 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:40:18.883774 22205 external_mini_cluster.cc:1428] Setting key 6919c3f50eb358f80d6e5859a9c85903
I20260810 03:40:18.886021 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:44325
--builtin_ntp_servers=127.21.175.84:34635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 03:40:18.887651 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.882366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:58609 (local address 127.21.175.126:44325)
0810 03:40:18.882519 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:18.882523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:18.883799 (+ 1276us) server_negotiation.cc:407] Connection header received
0810 03:40:18.885657 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:18.885663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:18.885706 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:18.885800 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:18.887117 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:18.887124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:18.887130 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:18.887148 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:18.887176 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:18.887190 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:18.887294 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:18.887407 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 03:40:18.887458 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:40:18.888876 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:18.882753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:44325 (local address 127.21.175.65:58609)
0810 03:40:18.883651 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:18.883698 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:18.885323 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:18.885945 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:18.885952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:18.886049 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:18.886819 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:40:18.886821 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:18.886866 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:40:18.886948 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:18.887271 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:18.887276 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:18.887281 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 03:40:18.887435 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 03:40:18.887717 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":120,"threads_started":1}
I20260810 03:40:18.890362 27294 heartbeater.cc:347] Connected to a master server at 127.21.175.126:44325
I20260810 03:40:18.890643 27294 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:18.891110 27294 heartbeater.cc:511] Master 127.21.175.126:44325 requested a full tablet report, sending...
I20260810 03:40:18.893002 27095 ts_manager.cc:194] Registered new tserver with Master: e485a606249a4a9188ab6ca47da8af06 (127.21.175.65:34811)
W20260810 03:40:19.014292 27298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:19.014588 27298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:19.014629 27298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:19.019048 27298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:19.019120 27298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:19.019181 27298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:40:19.024720 27298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:34635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:44325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:19.026019 27298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:19.027480 27298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:19.038244 27307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:19.038250 27303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:19.038295 27298 server_base.cc:1034] running on GCE node
W20260810 03:40:19.038250 27304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:19.039132 27298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:19.040161 27298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:19.041385 27298 hybrid_clock.cc:648] HybridClock initialized: now 1786333219041345 us; error 61 us; skew 500 ppm
I20260810 03:40:19.044164 27298 webserver.cc:460] Webserver started at http://127.21.175.66:41329/ using document root <none> and password file <none>
I20260810 03:40:19.045008 27298 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:19.045111 27298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:19.045374 27298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:19.048704 27298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "09bcb97852e5465a8433ecd2423f9d7b"
format_stamp: "Formatted at 2026-08-10 03:40:19 on dist-test-slave-ss4n"
server_key: "b7b5d7f812e57e3ae7b91e10f3a2e1f0"
server_key_iv: "c1264dddc475238b03a4cd3a798a58e7"
server_key_version: "encryptionkey@0"
I20260810 03:40:19.049582 27298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "09bcb97852e5465a8433ecd2423f9d7b"
format_stamp: "Formatted at 2026-08-10 03:40:19 on dist-test-slave-ss4n"
server_key: "b7b5d7f812e57e3ae7b91e10f3a2e1f0"
server_key_iv: "c1264dddc475238b03a4cd3a798a58e7"
server_key_version: "encryptionkey@0"
I20260810 03:40:19.055364 27298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 03:40:19.059170 27313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:19.060886 27298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:40:19.061049 27298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "09bcb97852e5465a8433ecd2423f9d7b"
format_stamp: "Formatted at 2026-08-10 03:40:19 on dist-test-slave-ss4n"
server_key: "b7b5d7f812e57e3ae7b91e10f3a2e1f0"
server_key_iv: "c1264dddc475238b03a4cd3a798a58e7"
server_key_version: "encryptionkey@0"
I20260810 03:40:19.061216 27298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:19.069267 27298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:19.070289 27298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:19.070518 27298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:19.071355 27298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:19.072886 27298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:19.072958 27298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:19.073028 27298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:19.073067 27298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:19.086474 27298 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:36361
I20260810 03:40:19.086503 27429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:36361 every 8 connection(s)
I20260810 03:40:19.087929 27298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:40:19.095358 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 27298
I20260810 03:40:19.095495 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:40:19.095726 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.090362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:35751 (local address 127.21.175.126:44325)
0810 03:40:19.090544 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:19.090550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:19.091893 (+ 1343us) server_negotiation.cc:407] Connection header received
0810 03:40:19.093427 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:19.093433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:19.093540 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:19.093671 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:19.095007 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:19.095013 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:19.095019 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:19.095039 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:19.095070 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:19.095084 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:19.095209 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:19.095474 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 03:40:19.095575 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:40:19.095813 22205 external_mini_cluster.cc:1428] Setting key 9d9ffdd238cf5410cd93343ad988cbda
I20260810 03:40:19.096937 27433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.090841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:44325 (local address 127.21.175.66:35751)
0810 03:40:19.091681 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:19.091741 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:19.093170 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:19.093857 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:19.093864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:19.093968 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:19.094687 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:40:19.094689 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:19.094735 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:40:19.094811 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:19.095267 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:19.095281 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:19.095291 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 03:40:19.095497 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 03:40:19.095807 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":163,"threads_started":1}
I20260810 03:40:19.098302 27430 heartbeater.cc:347] Connected to a master server at 127.21.175.126:44325
I20260810 03:40:19.098374 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:44325
--builtin_ntp_servers=127.21.175.84:34635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 03:40:19.098577 27430 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:19.099004 27430 heartbeater.cc:511] Master 127.21.175.126:44325 requested a full tablet report, sending...
I20260810 03:40:19.100178 27095 ts_manager.cc:194] Registered new tserver with Master: 09bcb97852e5465a8433ecd2423f9d7b (127.21.175.66:36361)
W20260810 03:40:19.222617 27434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:19.222913 27434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:19.222954 27434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:19.227131 27434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:19.227198 27434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:19.227258 27434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:40:19.232314 27434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:34635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:44325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:19.233579 27434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:19.234997 27434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:40:19.245702 27434 server_base.cc:1034] running on GCE node
W20260810 03:40:19.245625 27439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:19.245625 27442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:19.245628 27440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:19.246428 27434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:19.247184 27434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:19.248350 27434 hybrid_clock.cc:648] HybridClock initialized: now 1786333219248339 us; error 96 us; skew 500 ppm
I20260810 03:40:19.250687 27434 webserver.cc:460] Webserver started at http://127.21.175.67:39049/ using document root <none> and password file <none>
I20260810 03:40:19.251322 27434 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:19.251390 27434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:19.251618 27434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:19.253924 27434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "05a27bac5f644ad786470c5aea3149b1"
format_stamp: "Formatted at 2026-08-10 03:40:19 on dist-test-slave-ss4n"
server_key: "d1c690b668628a7aa14bdce65cd70428"
server_key_iv: "d649b1e35166606261d0c849b62ed9d7"
server_key_version: "encryptionkey@0"
I20260810 03:40:19.254530 27434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "05a27bac5f644ad786470c5aea3149b1"
format_stamp: "Formatted at 2026-08-10 03:40:19 on dist-test-slave-ss4n"
server_key: "d1c690b668628a7aa14bdce65cd70428"
server_key_iv: "d649b1e35166606261d0c849b62ed9d7"
server_key_version: "encryptionkey@0"
I20260810 03:40:19.258507 27434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:40:19.261143 27448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:19.262427 27434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:40:19.262528 27434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "05a27bac5f644ad786470c5aea3149b1"
format_stamp: "Formatted at 2026-08-10 03:40:19 on dist-test-slave-ss4n"
server_key: "d1c690b668628a7aa14bdce65cd70428"
server_key_iv: "d649b1e35166606261d0c849b62ed9d7"
server_key_version: "encryptionkey@0"
I20260810 03:40:19.262669 27434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:19.267601 27434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:19.268235 27434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:19.268431 27434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:19.268981 27434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:19.270155 27434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:19.270242 27434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:19.270329 27434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:19.270385 27434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:19.279644 27434 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:45601
I20260810 03:40:19.279664 27570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:45601 every 8 connection(s)
I20260810 03:40:19.280650 27434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:40:19.286931 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 27434
I20260810 03:40:19.287042 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:40:19.287366 22205 external_mini_cluster.cc:1428] Setting key fbecba9c4248a0508b61f6cc76fd2e02
I20260810 03:40:19.287684 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.282854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:56843 (local address 127.21.175.126:44325)
0810 03:40:19.283055 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:19.283059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:19.284254 (+ 1195us) server_negotiation.cc:407] Connection header received
0810 03:40:19.285755 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:19.285759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:19.285807 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:19.285900 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:19.287164 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:19.287168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:19.287170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:19.287185 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:19.287217 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:19.287226 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:19.287329 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:19.287494 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 03:40:19.287544 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260810 03:40:19.288888 27573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.283288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:44325 (local address 127.21.175.67:56843)
0810 03:40:19.284079 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:19.284124 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:19.285451 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:19.286100 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:19.286108 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:19.286203 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:19.286867 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:40:19.286869 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:19.286912 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:40:19.286989 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:19.287321 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:19.287330 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:19.287338 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 03:40:19.287540 (+ 202us) client_negotiation.cc:240] Negotiation successful
0810 03:40:19.287799 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":146,"threads_started":1}
I20260810 03:40:19.290158 27571 heartbeater.cc:347] Connected to a master server at 127.21.175.126:44325
I20260810 03:40:19.290386 27571 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:19.290778 27571 heartbeater.cc:511] Master 127.21.175.126:44325 requested a full tablet report, sending...
I20260810 03:40:19.291821 27095 ts_manager.cc:194] Registered new tserver with Master: 05a27bac5f644ad786470c5aea3149b1 (127.21.175.67:45601)
I20260810 03:40:19.302525 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:40:19.306362 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.304724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57560 (local address 127.21.175.126:44325)
0810 03:40:19.304861 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:19.304865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:19.305031 (+ 166us) server_negotiation.cc:407] Connection header received
0810 03:40:19.305129 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:19.305131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:19.305170 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:19.305234 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:19.306024 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:19.306028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:19.306030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:19.306042 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:19.306063 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:19.306072 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:19.306150 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:19.306215 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 03:40:19.306250 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:40:19.310079 27095 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:40:19.313273 27095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:40:19.326265 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.322933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:45601 (local address 127.0.0.1:40466)
0810 03:40:19.323324 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:19.323384 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:19.323492 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:19.324543 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:19.324551 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:19.324646 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:19.325245 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:40:19.325250 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:19.325274 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:40:19.325352 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:19.325880 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:19.325892 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:19.325899 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:40:19.325994 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 03:40:19.326084 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":102,"threads_started":1}
I20260810 03:40:19.326495 27588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.323827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.21.175.65:34811)
0810 03:40:19.324469 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:19.324482 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:19.324503 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:40:19.324596 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:19.324602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:19.324773 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:19.325198 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:19.325926 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:19.325932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:19.325943 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:19.325954 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:19.325996 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:19.326023 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:19.326125 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:19.326234 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 03:40:19.326313 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":148,"threads_started":1}
I20260810 03:40:19.327273 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.323406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:34811 (local address 127.0.0.1:41378)
0810 03:40:19.323700 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:19.323715 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:19.324207 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:19.324944 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:19.324947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:19.324972 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:19.325410 (+ 438us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:40:19.325412 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:19.325423 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:40:19.325454 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:19.326101 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:19.326105 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:19.326108 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 03:40:19.326197 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 03:40:19.326235 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":106,"threads_started":1}
I20260810 03:40:19.327306 27587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.323697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:36361 (local address 127.0.0.1:51120)
0810 03:40:19.325006 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:19.325026 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:19.325135 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:19.325820 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:19.325823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:19.325838 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:19.326678 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:40:19.326682 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:19.326695 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:40:19.326722 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:19.327060 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:19.327063 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:19.327066 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 03:40:19.327151 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 03:40:19.327186 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1240,"thread_start_us":91,"threads_started":1}
I20260810 03:40:19.327410 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.324116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51120 (local address 127.21.175.66:36361)
0810 03:40:19.324539 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:19.324551 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:19.325391 (+ 840us) server_negotiation.cc:407] Connection header received
0810 03:40:19.325526 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:19.325532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:19.325695 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:19.325835 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:19.326848 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:19.326855 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:19.326864 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:19.326876 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:19.326920 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:19.326950 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:19.327065 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:19.327167 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 03:40:19.327240 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":117,"threads_started":1}
I20260810 03:40:19.327713 27585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.323073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.21.175.67:45601)
0810 03:40:19.323988 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:19.324003 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:19.324044 (+ 41us) server_negotiation.cc:407] Connection header received
0810 03:40:19.324103 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:19.324109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:19.324325 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:19.324515 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:19.325634 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:19.325643 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:19.325659 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:19.325673 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:19.325722 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:19.325760 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:19.327079 (+ 1319us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:19.327129 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 03:40:19.327204 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":127,"threads_started":1}
I20260810 03:40:19.330255 27503 tablet_service.cc:1467] Processing CreateTablet for tablet 455e4c8f8e9445f39b617de298f1c117 (DEFAULT_TABLE table=test-table [id=dcb89b4c08b646c6ac28ce003f937bc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:19.330255 27344 tablet_service.cc:1467] Processing CreateTablet for tablet 455e4c8f8e9445f39b617de298f1c117 (DEFAULT_TABLE table=test-table [id=dcb89b4c08b646c6ac28ce003f937bc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:19.330261 27226 tablet_service.cc:1467] Processing CreateTablet for tablet 455e4c8f8e9445f39b617de298f1c117 (DEFAULT_TABLE table=test-table [id=dcb89b4c08b646c6ac28ce003f937bc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:19.331384 27226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 455e4c8f8e9445f39b617de298f1c117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:19.331405 27344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 455e4c8f8e9445f39b617de298f1c117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:19.331404 27503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 455e4c8f8e9445f39b617de298f1c117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:19.338151 27590 tablet_bootstrap.cc:492] T 455e4c8f8e9445f39b617de298f1c117 P e485a606249a4a9188ab6ca47da8af06: Bootstrap starting.
I20260810 03:40:19.339551 27592 tablet_bootstrap.cc:492] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1: Bootstrap starting.
I20260810 03:40:19.341130 27590 tablet_bootstrap.cc:654] T 455e4c8f8e9445f39b617de298f1c117 P e485a606249a4a9188ab6ca47da8af06: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:19.341759 27592 tablet_bootstrap.cc:654] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:19.341919 27591 tablet_bootstrap.cc:492] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b: Bootstrap starting.
I20260810 03:40:19.342173 27590 log.cc:824] T 455e4c8f8e9445f39b617de298f1c117 P e485a606249a4a9188ab6ca47da8af06: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:19.342625 27592 log.cc:824] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:19.344201 27590 tablet_bootstrap.cc:492] T 455e4c8f8e9445f39b617de298f1c117 P e485a606249a4a9188ab6ca47da8af06: No bootstrap required, opened a new log
I20260810 03:40:19.344208 27591 tablet_bootstrap.cc:654] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:19.344436 27590 ts_tablet_manager.cc:1397] T 455e4c8f8e9445f39b617de298f1c117 P e485a606249a4a9188ab6ca47da8af06: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:40:19.344659 27592 tablet_bootstrap.cc:492] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1: No bootstrap required, opened a new log
I20260810 03:40:19.344844 27592 ts_tablet_manager.cc:1397] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 03:40:19.345431 27591 log.cc:824] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:19.348338 27590 raft_consensus.cc:348] T 455e4c8f8e9445f39b617de298f1c117 P e485a606249a4a9188ab6ca47da8af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a27bac5f644ad786470c5aea3149b1" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 45601 } } peers { permanent_uuid: "09bcb97852e5465a8433ecd2423f9d7b" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 36361 } } peers { permanent_uuid: "e485a606249a4a9188ab6ca47da8af06" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34811 } }
I20260810 03:40:19.348438 27592 raft_consensus.cc:348] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a27bac5f644ad786470c5aea3149b1" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 45601 } } peers { permanent_uuid: "09bcb97852e5465a8433ecd2423f9d7b" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 36361 } } peers { permanent_uuid: "e485a606249a4a9188ab6ca47da8af06" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34811 } }
I20260810 03:40:19.348627 27590 raft_consensus.cc:374] T 455e4c8f8e9445f39b617de298f1c117 P e485a606249a4a9188ab6ca47da8af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:19.348667 27592 raft_consensus.cc:374] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:19.348701 27590 raft_consensus.cc:729] T 455e4c8f8e9445f39b617de298f1c117 P e485a606249a4a9188ab6ca47da8af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e485a606249a4a9188ab6ca47da8af06, State: Initialized, Role: FOLLOWER
I20260810 03:40:19.348726 27592 raft_consensus.cc:729] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05a27bac5f644ad786470c5aea3149b1, State: Initialized, Role: FOLLOWER
I20260810 03:40:19.349217 27590 consensus_queue.cc:260] T 455e4c8f8e9445f39b617de298f1c117 P e485a606249a4a9188ab6ca47da8af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a27bac5f644ad786470c5aea3149b1" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 45601 } } peers { permanent_uuid: "09bcb97852e5465a8433ecd2423f9d7b" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 36361 } } peers { permanent_uuid: "e485a606249a4a9188ab6ca47da8af06" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34811 } }
I20260810 03:40:19.349223 27592 consensus_queue.cc:260] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a27bac5f644ad786470c5aea3149b1" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 45601 } } peers { permanent_uuid: "09bcb97852e5465a8433ecd2423f9d7b" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 36361 } } peers { permanent_uuid: "e485a606249a4a9188ab6ca47da8af06" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34811 } }
I20260810 03:40:19.349941 27294 heartbeater.cc:503] Master 127.21.175.126:44325 was elected leader, sending a full tablet report...
I20260810 03:40:19.350270 27590 ts_tablet_manager.cc:1428] T 455e4c8f8e9445f39b617de298f1c117 P e485a606249a4a9188ab6ca47da8af06: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 03:40:19.350288 27592 ts_tablet_manager.cc:1428] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 03:40:19.350592 27571 heartbeater.cc:503] Master 127.21.175.126:44325 was elected leader, sending a full tablet report...
I20260810 03:40:19.351384 27591 tablet_bootstrap.cc:492] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b: No bootstrap required, opened a new log
I20260810 03:40:19.351601 27591 ts_tablet_manager.cc:1397] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260810 03:40:19.356227 27591 raft_consensus.cc:348] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a27bac5f644ad786470c5aea3149b1" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 45601 } } peers { permanent_uuid: "09bcb97852e5465a8433ecd2423f9d7b" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 36361 } } peers { permanent_uuid: "e485a606249a4a9188ab6ca47da8af06" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34811 } }
I20260810 03:40:19.356470 27591 raft_consensus.cc:374] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:19.356503 27591 raft_consensus.cc:729] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bcb97852e5465a8433ecd2423f9d7b, State: Initialized, Role: FOLLOWER
I20260810 03:40:19.356958 27591 consensus_queue.cc:260] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a27bac5f644ad786470c5aea3149b1" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 45601 } } peers { permanent_uuid: "09bcb97852e5465a8433ecd2423f9d7b" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 36361 } } peers { permanent_uuid: "e485a606249a4a9188ab6ca47da8af06" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34811 } }
I20260810 03:40:19.357744 27430 heartbeater.cc:503] Master 127.21.175.126:44325 was elected leader, sending a full tablet report...
I20260810 03:40:19.357937 27591 ts_tablet_manager.cc:1428] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260810 03:40:19.381453 27295 tablet.cc:2369] T 455e4c8f8e9445f39b617de298f1c117 P e485a606249a4a9188ab6ca47da8af06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:40:19.440021 27597 raft_consensus.cc:482] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:19.440236 27597 raft_consensus.cc:504] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a27bac5f644ad786470c5aea3149b1" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 45601 } } peers { permanent_uuid: "09bcb97852e5465a8433ecd2423f9d7b" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 36361 } } peers { permanent_uuid: "e485a606249a4a9188ab6ca47da8af06" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34811 } }
I20260810 03:40:19.441751 27597 leader_election.cc:288] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09bcb97852e5465a8433ecd2423f9d7b (127.21.175.66:36361), e485a606249a4a9188ab6ca47da8af06 (127.21.175.65:34811)
I20260810 03:40:19.444162 27573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.442047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:34811 (local address 127.21.175.67:43229)
0810 03:40:19.442229 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:19.442246 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:19.442341 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:19.442687 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:19.442691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:19.442709 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:19.443361 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:40:19.443364 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:19.443380 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:40:19.443420 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:19.443827 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:19.443830 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:19.443834 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 03:40:19.443941 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 03:40:19.443986 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 03:40:19.444197 27588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.442205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:43229 (local address 127.21.175.65:34811)
0810 03:40:19.442344 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:19.442348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:19.442369 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:40:19.442425 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:19.442428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:19.442471 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:19.442580 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:19.443623 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:19.443627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:19.443630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:19.443645 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:19.443668 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:19.443679 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:19.443799 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:19.443992 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 03:40:19.444044 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:40:19.444329 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.442309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:55693 (local address 127.21.175.66:36361)
0810 03:40:19.442446 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:19.442450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:19.442635 (+ 185us) server_negotiation.cc:407] Connection header received
0810 03:40:19.442699 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:19.442702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:19.442751 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:19.442862 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:19.443846 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:19.443850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:19.443853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:19.443869 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:19.443896 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:19.443919 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:19.444008 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:19.444129 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 03:40:19.444173 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:40:19.444355 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.442048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:36361 (local address 127.21.175.67:55693)
0810 03:40:19.442482 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:19.442496 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:19.442599 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:19.442934 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:19.442940 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:19.442959 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:19.443632 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:40:19.443635 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:19.443647 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:40:19.443675 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:19.444014 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:19.444019 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:19.444022 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 03:40:19.444114 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 03:40:19.444148 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":31,"thread_start_us":143,"threads_started":1}
I20260810 03:40:19.444940 27246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "455e4c8f8e9445f39b617de298f1c117" candidate_uuid: "05a27bac5f644ad786470c5aea3149b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e485a606249a4a9188ab6ca47da8af06" is_pre_election: true
I20260810 03:40:19.445151 27384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "455e4c8f8e9445f39b617de298f1c117" candidate_uuid: "05a27bac5f644ad786470c5aea3149b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09bcb97852e5465a8433ecd2423f9d7b" is_pre_election: true
I20260810 03:40:19.445343 27246 raft_consensus.cc:2454] T 455e4c8f8e9445f39b617de298f1c117 P e485a606249a4a9188ab6ca47da8af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05a27bac5f644ad786470c5aea3149b1 in term 0.
I20260810 03:40:19.445396 27384 raft_consensus.cc:2454] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05a27bac5f644ad786470c5aea3149b1 in term 0.
I20260810 03:40:19.445912 27450 leader_election.cc:302] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05a27bac5f644ad786470c5aea3149b1, 09bcb97852e5465a8433ecd2423f9d7b, e485a606249a4a9188ab6ca47da8af06; no voters:
I20260810 03:40:19.446211 27597 raft_consensus.cc:2781] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:40:19.446302 27597 raft_consensus.cc:482] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:40:19.446365 27597 raft_consensus.cc:3037] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:19.447628 27597 raft_consensus.cc:504] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a27bac5f644ad786470c5aea3149b1" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 45601 } } peers { permanent_uuid: "09bcb97852e5465a8433ecd2423f9d7b" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 36361 } } peers { permanent_uuid: "e485a606249a4a9188ab6ca47da8af06" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34811 } }
I20260810 03:40:19.448019 27597 leader_election.cc:288] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [CANDIDATE]: Term 1 election: Requested vote from peers 09bcb97852e5465a8433ecd2423f9d7b (127.21.175.66:36361), e485a606249a4a9188ab6ca47da8af06 (127.21.175.65:34811)
I20260810 03:40:19.448419 27246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "455e4c8f8e9445f39b617de298f1c117" candidate_uuid: "05a27bac5f644ad786470c5aea3149b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e485a606249a4a9188ab6ca47da8af06"
I20260810 03:40:19.448428 27384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "455e4c8f8e9445f39b617de298f1c117" candidate_uuid: "05a27bac5f644ad786470c5aea3149b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09bcb97852e5465a8433ecd2423f9d7b"
I20260810 03:40:19.448534 27246 raft_consensus.cc:3037] T 455e4c8f8e9445f39b617de298f1c117 P e485a606249a4a9188ab6ca47da8af06 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:19.448598 27384 raft_consensus.cc:3037] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:19.449829 27246 raft_consensus.cc:2454] T 455e4c8f8e9445f39b617de298f1c117 P e485a606249a4a9188ab6ca47da8af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05a27bac5f644ad786470c5aea3149b1 in term 1.
I20260810 03:40:19.449913 27384 raft_consensus.cc:2454] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05a27bac5f644ad786470c5aea3149b1 in term 1.
I20260810 03:40:19.450176 27450 leader_election.cc:302] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a27bac5f644ad786470c5aea3149b1, e485a606249a4a9188ab6ca47da8af06; no voters:
I20260810 03:40:19.450399 27597 raft_consensus.cc:2781] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:19.450598 27597 raft_consensus.cc:686] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [term 1 LEADER]: Becoming Leader. State: Replica: 05a27bac5f644ad786470c5aea3149b1, State: Running, Role: LEADER
I20260810 03:40:19.450973 27597 consensus_queue.cc:237] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a27bac5f644ad786470c5aea3149b1" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 45601 } } peers { permanent_uuid: "09bcb97852e5465a8433ecd2423f9d7b" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 36361 } } peers { permanent_uuid: "e485a606249a4a9188ab6ca47da8af06" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34811 } }
I20260810 03:40:19.454604 27094 catalog_manager.cc:5495] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05a27bac5f644ad786470c5aea3149b1 (127.21.175.67). New cstate: current_term: 1 leader_uuid: "05a27bac5f644ad786470c5aea3149b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a27bac5f644ad786470c5aea3149b1" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 45601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09bcb97852e5465a8433ecd2423f9d7b" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 36361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e485a606249a4a9188ab6ca47da8af06" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34811 } health_report { overall_health: UNKNOWN } } }
I20260810 03:40:19.482802 27585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.481059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40482 (local address 127.21.175.67:45601)
0810 03:40:19.481196 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:19.481200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:19.481213 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:40:19.481360 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:19.481362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:19.481429 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:19.481585 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:19.482333 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:19.482336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:19.482339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:19.482351 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:19.482372 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:19.482381 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:19.482461 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:19.482651 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 03:40:19.482691 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:40:19.489307 27246 raft_consensus.cc:1261] T 455e4c8f8e9445f39b617de298f1c117 P e485a606249a4a9188ab6ca47da8af06 [term 1 FOLLOWER]: Refusing update from remote peer 05a27bac5f644ad786470c5aea3149b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:19.489307 27384 raft_consensus.cc:1261] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b [term 1 FOLLOWER]: Refusing update from remote peer 05a27bac5f644ad786470c5aea3149b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:19.490361 27597 consensus_queue.cc:1035] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e485a606249a4a9188ab6ca47da8af06" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:19.490664 27600 consensus_queue.cc:1035] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09bcb97852e5465a8433ecd2423f9d7b" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 36361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:19.501545 27604 mvcc.cc:205] Tried to move back new op lower bound from 7316820867019997184 to 7316820866873876480. Current Snapshot: MvccSnapshot[applied={T|T < 7316820867019997184 or (T in {7316820867019997184})}]
I20260810 03:40:19.508464 27094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:19.508680 27094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:19.511521 27094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd7a07b10bfa4a08a37ac916db69bac4: Sending DeleteTablet for 3 replicas of tablet 455e4c8f8e9445f39b617de298f1c117
I20260810 03:40:19.512466 27344 tablet_service.cc:1514] Processing DeleteTablet for tablet 455e4c8f8e9445f39b617de298f1c117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:19 UTC) from {username='slave'} at 127.0.0.1:51120
I20260810 03:40:19.512456 27503 tablet_service.cc:1514] Processing DeleteTablet for tablet 455e4c8f8e9445f39b617de298f1c117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:19 UTC) from {username='slave'} at 127.0.0.1:40466
I20260810 03:40:19.512614 27226 tablet_service.cc:1514] Processing DeleteTablet for tablet 455e4c8f8e9445f39b617de298f1c117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:19 UTC) from {username='slave'} at 127.0.0.1:41378
I20260810 03:40:19.513029 27613 tablet_replica.cc:330] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b: stopping tablet replica
I20260810 03:40:19.513356 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 27159
I20260810 03:40:19.523550 27613 raft_consensus.cc:2229] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:19.524082 27613 raft_consensus.cc:2258] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:19.524379 27614 tablet_replica.cc:330] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1: stopping tablet replica
I20260810 03:40:19.524717 27614 raft_consensus.cc:2229] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:40:19.525080 27614 raft_consensus.cc:2258] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:19.526726 27613 ts_tablet_manager.cc:1905] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:19.529284 27613 ts_tablet_manager.cc:1918] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:40:19.529521 27613 log.cc:1196] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/455e4c8f8e9445f39b617de298f1c117
I20260810 03:40:19.529970 27613 ts_tablet_manager.cc:1939] T 455e4c8f8e9445f39b617de298f1c117 P 09bcb97852e5465a8433ecd2423f9d7b: Deleting consensus metadata
I20260810 03:40:19.530325 27614 ts_tablet_manager.cc:1905] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:19.531651 27079 catalog_manager.cc:4841] TS 09bcb97852e5465a8433ecd2423f9d7b (127.21.175.66:36361): tablet 455e4c8f8e9445f39b617de298f1c117 (table test-table [id=dcb89b4c08b646c6ac28ce003f937bc5]) successfully deleted
W20260810 03:40:19.531949 27080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.175.65:34811 (error 108)
I20260810 03:40:19.532295 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 27298
I20260810 03:40:19.534900 27587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.534471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:34811 (local address 127.0.0.1:41384)
0810 03:40:19.534657 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:19.534772 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:34811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 03:40:19.535156 27080 catalog_manager.cc:4584] TS e485a606249a4a9188ab6ca47da8af06 (127.21.175.65:34811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 455e4c8f8e9445f39b617de298f1c117: Network error: Client connection negotiation failed: client connection to 127.21.175.65:34811: connect: Connection refused (error 111)
I20260810 03:40:19.535228 27614 ts_tablet_manager.cc:1918] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:40:19.535346 27614 log.cc:1196] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/455e4c8f8e9445f39b617de298f1c117
I20260810 03:40:19.535666 27614 ts_tablet_manager.cc:1939] T 455e4c8f8e9445f39b617de298f1c117 P 05a27bac5f644ad786470c5aea3149b1: Deleting consensus metadata
I20260810 03:40:19.536844 27079 catalog_manager.cc:4841] TS 05a27bac5f644ad786470c5aea3149b1 (127.21.175.67:45601): tablet 455e4c8f8e9445f39b617de298f1c117 (table test-table [id=dcb89b4c08b646c6ac28ce003f937bc5]) successfully deleted
I20260810 03:40:19.544764 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 27434
I20260810 03:40:19.554545 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 27063
2026-08-10T03:40:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1158 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 03:40:19.580505 22205 mini_oidc.cc:152] Starting JWKS server
I20260810 03:40:19.582542 22205 webserver.cc:460] Webserver started at https://127.0.0.1:46689/ using document root <none> and password file <none>
I20260810 03:40:19.582756 22205 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 03:40:19.583061 22205 webserver.cc:460] Webserver started at http://127.0.0.1:37851/ using document root <none> and password file <none>
2026-08-10T03:40:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:19Z Disabled control of system clock
I20260810 03:40:19.593840 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:33059
--webserver_interface=127.21.175.126
--webserver_port=0
--builtin_ntp_servers=127.21.175.84:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:33059
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46689/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/testchainca.cert with env {}
W20260810 03:40:19.715130 27623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:19.715442 27623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:19.715492 27623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:19.719772 27623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:19.719880 27623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:19.719913 27623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:19.719933 27623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:19.719949 27623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 03:40:19.719967 27623 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 03:40:19.719981 27623 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46689/jwks.json
I20260810 03:40:19.725972 27623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:33059
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:33059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46689/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:19.727123 27623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:19.728148 27623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:19.737180 27629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:19.737180 27632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:19.737186 27628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:19.737910 27623 server_base.cc:1034] running on GCE node
I20260810 03:40:19.738616 27623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:19.739796 27623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:19.740998 27623 hybrid_clock.cc:648] HybridClock initialized: now 1786333219740975 us; error 39 us; skew 500 ppm
I20260810 03:40:19.743199 27623 webserver.cc:460] Webserver started at http://127.21.175.126:39387/ using document root <none> and password file <none>
I20260810 03:40:19.743816 27623 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:19.743912 27623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:19.744151 27623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:19.746467 27623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "be119767659440d08b5828b8593c32a9"
format_stamp: "Formatted at 2026-08-10 03:40:19 on dist-test-slave-ss4n"
server_key: "da4a6e48dc8f82f865867b71871a07b0"
server_key_iv: "a8c7aeb33ecf09ab01449e2c6713a010"
server_key_version: "encryptionkey@0"
I20260810 03:40:19.747278 27623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "be119767659440d08b5828b8593c32a9"
format_stamp: "Formatted at 2026-08-10 03:40:19 on dist-test-slave-ss4n"
server_key: "da4a6e48dc8f82f865867b71871a07b0"
server_key_iv: "a8c7aeb33ecf09ab01449e2c6713a010"
server_key_version: "encryptionkey@0"
I20260810 03:40:19.751573 27623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:40:19.754263 27639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:19.755580 27623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:19.755734 27623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "be119767659440d08b5828b8593c32a9"
format_stamp: "Formatted at 2026-08-10 03:40:19 on dist-test-slave-ss4n"
server_key: "da4a6e48dc8f82f865867b71871a07b0"
server_key_iv: "a8c7aeb33ecf09ab01449e2c6713a010"
server_key_version: "encryptionkey@0"
I20260810 03:40:19.755861 27623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:19.775213 27623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:19.776031 27623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:19.776252 27623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:19.783952 27623 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:33059
I20260810 03:40:19.783975 27696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:33059 every 8 connection(s)
I20260810 03:40:19.785048 27623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:19.788401 27697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:19.791375 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 27623
I20260810 03:40:19.791532 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:40:19.791850 22205 external_mini_cluster.cc:1428] Setting key f0604462f6a5a8d24fac515bad302d9a
I20260810 03:40:19.795274 27697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9: Bootstrap starting.
I20260810 03:40:19.797688 27697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:19.798458 27697 log.cc:824] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:19.800477 27697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9: No bootstrap required, opened a new log
I20260810 03:40:19.801728 27701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.793465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.21.175.126:33059)
0810 03:40:19.794215 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:19.794231 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:19.794269 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:40:19.794900 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:19.794917 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:19.795330 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:19.795725 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:19.796199 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:19.797199 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:19.798238 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:19.798779 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:19.798951 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:19.799832 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:19.799843 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:19.799857 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:19.800300 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:19.800360 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:19.800381 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:19.800493 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:19.800635 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 03:40:19.800907 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":151,"threads_started":1}
I20260810 03:40:19.804867 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:40:19.805024 27697 raft_consensus.cc:348] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be119767659440d08b5828b8593c32a9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 33059 } }
I20260810 03:40:19.805299 27697 raft_consensus.cc:374] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:19.805392 27697 raft_consensus.cc:729] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be119767659440d08b5828b8593c32a9, State: Initialized, Role: FOLLOWER
I20260810 03:40:19.805932 27697 consensus_queue.cc:260] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be119767659440d08b5828b8593c32a9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 33059 } }
I20260810 03:40:19.806119 27697 raft_consensus.cc:388] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:19.806195 27697 raft_consensus.cc:482] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:19.806299 27697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:19.807423 27697 raft_consensus.cc:504] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be119767659440d08b5828b8593c32a9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 33059 } }
I20260810 03:40:19.807816 27697 leader_election.cc:302] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be119767659440d08b5828b8593c32a9; no voters:
I20260810 03:40:19.808193 27697 leader_election.cc:288] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:40:19.808336 27703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:19.808561 27703 raft_consensus.cc:686] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [term 1 LEADER]: Becoming Leader. State: Replica: be119767659440d08b5828b8593c32a9, State: Running, Role: LEADER
I20260810 03:40:19.808934 27703 consensus_queue.cc:237] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be119767659440d08b5828b8593c32a9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 33059 } }
I20260810 03:40:19.809700 27697 sys_catalog.cc:564] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:19.810830 27705 sys_catalog.cc:455] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be119767659440d08b5828b8593c32a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be119767659440d08b5828b8593c32a9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 33059 } } }
I20260810 03:40:19.810981 27705 sys_catalog.cc:458] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:19.811280 27704 sys_catalog.cc:455] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be119767659440d08b5828b8593c32a9. Latest consensus state: current_term: 1 leader_uuid: "be119767659440d08b5828b8593c32a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be119767659440d08b5828b8593c32a9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 33059 } } }
I20260810 03:40:19.811376 27704 sys_catalog.cc:458] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9 [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:19.811422 27713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:19.814747 27713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:19.821432 27713 catalog_manager.cc:1347] Generated new cluster ID: 6465ac477c7242f8856a4c3cdd4ac9ee
I20260810 03:40:19.821555 27713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:19.834219 27713 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:40:19.835345 27713 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:19.847131 27713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be119767659440d08b5828b8593c32a9: Generated new TSK 0
I20260810 03:40:19.847955 27713 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 03:40:19.894912 27730 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 03:40:19.895104 27730 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 03:40:19.895750 27701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:19.870456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.21.175.126:33059)
0810 03:40:19.870636 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:19.870641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:19.870850 (+ 209us) server_negotiation.cc:407] Connection header received
0810 03:40:19.871133 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:19.871136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:19.871190 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:19.871306 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0810 03:40:19.871592 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:19.872168 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:19.873051 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:19.873298 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:19.873411 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:19.873842 (+ 431us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 03:40:19.895087 (+ 21245us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46689/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 03:40:19.895544 (+ 457us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47172: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46689/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 03:40:19.895843 27701 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47172: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46689/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 03:40:19.897346 27729 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:19.870353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:33059 (local address 127.0.0.1:47172)
0810 03:40:19.870751 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:19.870766 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:19.870936 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:19.871337 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:19.871340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:19.871349 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 03:40:19.871477 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:19.871482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:19.872290 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:19.872294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:19.872925 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:19.872933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:19.873433 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:19.873437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:19.873566 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:19.873628 (+ 62us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 03:40:19.895397 (+ 21769us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46689/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 03:40:19.895691 (+ 294us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.175.126:33059: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46689/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":127,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.175.126:33059: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46689/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 03:40:19.898270 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 27623
2026-08-10T03:40:19Z chronyd exiting
I20260810 03:40:19.941809 22205 test_util.cc:175] -----------------------------------------------
I20260810 03:40:19.941938 22205 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333190020696-22205-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (363 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 03:40:19.943142 22205 mini_oidc.cc:152] Starting JWKS server
I20260810 03:40:19.944036 22205 webserver.cc:460] Webserver started at https://127.0.0.1:38163/ using document root <none> and password file <none>
I20260810 03:40:19.944161 22205 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 03:40:19.944489 22205 webserver.cc:460] Webserver started at http://127.0.0.1:34001/ using document root <none> and password file <none>
2026-08-10T03:40:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:19Z Disabled control of system clock
I20260810 03:40:19.954746 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:32971
--webserver_interface=127.21.175.126
--webserver_port=0
--builtin_ntp_servers=127.21.175.84:38135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:32971
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38163/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/testchainca.cert with env {}
W20260810 03:40:20.071403 27736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:20.071785 27736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:20.071870 27736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:20.076071 27736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:20.076201 27736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:20.076231 27736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:20.076270 27736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:20.076306 27736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 03:40:20.076340 27736 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 03:40:20.076372 27736 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38163/jwks.json
I20260810 03:40:20.084188 27736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:38135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:32971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:32971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38163/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:20.085429 27736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:20.086558 27736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:20.096021 27745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:20.096146 27736 server_base.cc:1034] running on GCE node
W20260810 03:40:20.096024 27743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:20.096021 27742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:20.096995 27736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:20.098263 27736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:20.099444 27736 hybrid_clock.cc:648] HybridClock initialized: now 1786333220099437 us; error 42 us; skew 500 ppm
I20260810 03:40:20.101441 27736 webserver.cc:460] Webserver started at http://127.21.175.126:41479/ using document root <none> and password file <none>
I20260810 03:40:20.102123 27736 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:20.102203 27736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:20.102432 27736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:20.104642 27736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "0fbf857737e345c69b59fdcbe2e6d891"
format_stamp: "Formatted at 2026-08-10 03:40:20 on dist-test-slave-ss4n"
server_key: "174a52d2f39421f09ea99e461e508ff3"
server_key_iv: "a422fa67fd2db8bae4e7bb2c360e94be"
server_key_version: "encryptionkey@0"
I20260810 03:40:20.105237 27736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "0fbf857737e345c69b59fdcbe2e6d891"
format_stamp: "Formatted at 2026-08-10 03:40:20 on dist-test-slave-ss4n"
server_key: "174a52d2f39421f09ea99e461e508ff3"
server_key_iv: "a422fa67fd2db8bae4e7bb2c360e94be"
server_key_version: "encryptionkey@0"
I20260810 03:40:20.109640 27736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 03:40:20.112342 27751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:20.113834 27736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:20.113989 27736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "0fbf857737e345c69b59fdcbe2e6d891"
format_stamp: "Formatted at 2026-08-10 03:40:20 on dist-test-slave-ss4n"
server_key: "174a52d2f39421f09ea99e461e508ff3"
server_key_iv: "a422fa67fd2db8bae4e7bb2c360e94be"
server_key_version: "encryptionkey@0"
I20260810 03:40:20.114123 27736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:20.128842 27736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:20.129765 27736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:20.129982 27736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:20.137629 27736 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:32971
I20260810 03:40:20.137674 27808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:32971 every 8 connection(s)
I20260810 03:40:20.138847 27736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:20.141665 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 27736
I20260810 03:40:20.141798 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:40:20.142129 22205 external_mini_cluster.cc:1428] Setting key 3d6078f8d9be0bdab483b46c347aa5d9
I20260810 03:40:20.142838 27809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:20.149430 27809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891: Bootstrap starting.
I20260810 03:40:20.152041 27813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:20.143581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.21.175.126:32971)
0810 03:40:20.144336 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:20.144353 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:20.144392 (+ 39us) server_negotiation.cc:407] Connection header received
0810 03:40:20.145110 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:20.145128 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:20.145644 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:20.145989 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:20.146443 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.147417 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.148377 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.148881 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.149062 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:20.150122 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:20.150127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:20.150142 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:20.150543 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:20.150629 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:20.150652 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:20.150733 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:20.150941 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 03:40:20.151263 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":182,"threads_started":1}
I20260810 03:40:20.152328 27809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:20.153149 27809 log.cc:824] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:20.155081 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:40:20.155481 27809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891: No bootstrap required, opened a new log
I20260810 03:40:20.159386 27809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fbf857737e345c69b59fdcbe2e6d891" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 32971 } }
I20260810 03:40:20.159690 27809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:20.159785 27809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbf857737e345c69b59fdcbe2e6d891, State: Initialized, Role: FOLLOWER
I20260810 03:40:20.160239 27809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fbf857737e345c69b59fdcbe2e6d891" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 32971 } }
I20260810 03:40:20.160403 27809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:20.160492 27809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:20.160624 27809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:20.161762 27809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fbf857737e345c69b59fdcbe2e6d891" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 32971 } }
I20260810 03:40:20.162204 27809 leader_election.cc:302] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fbf857737e345c69b59fdcbe2e6d891; no voters:
I20260810 03:40:20.162582 27809 leader_election.cc:288] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:40:20.162739 27815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:20.163013 27815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [term 1 LEADER]: Becoming Leader. State: Replica: 0fbf857737e345c69b59fdcbe2e6d891, State: Running, Role: LEADER
I20260810 03:40:20.163400 27815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fbf857737e345c69b59fdcbe2e6d891" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 32971 } }
I20260810 03:40:20.164003 27809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:20.165766 27817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fbf857737e345c69b59fdcbe2e6d891. Latest consensus state: current_term: 1 leader_uuid: "0fbf857737e345c69b59fdcbe2e6d891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fbf857737e345c69b59fdcbe2e6d891" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 32971 } } }
I20260810 03:40:20.165900 27817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:20.166323 27816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fbf857737e345c69b59fdcbe2e6d891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fbf857737e345c69b59fdcbe2e6d891" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 32971 } } }
I20260810 03:40:20.166584 27816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891 [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:20.166823 27824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:20.169682 27824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:20.176083 27824 catalog_manager.cc:1347] Generated new cluster ID: 0762a9ff22714f498c001add1a1870e4
I20260810 03:40:20.176175 27824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:20.183164 27824 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:40:20.184485 27824 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:20.197614 27824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fbf857737e345c69b59fdcbe2e6d891: Generated new TSK 0
I20260810 03:40:20.198475 27824 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 03:40:20.233809 27841 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 03:40:20.233919 27841 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 03:40:20.234694 27813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:20.217571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.21.175.126:32971)
0810 03:40:20.217728 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:20.217733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:20.218037 (+ 304us) server_negotiation.cc:407] Connection header received
0810 03:40:20.218220 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:20.218225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:20.218285 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:20.218368 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0810 03:40:20.218790 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.219422 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.220708 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.220936 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.221068 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:20.221428 (+ 360us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 03:40:20.234273 (+ 12845us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38163/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 03:40:20.234536 (+ 263us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54608: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38163/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 03:40:20.234797 27813 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54608: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38163/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 03:40:20.234962 27840 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:20.217455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:32971 (local address 127.0.0.1:54608)
0810 03:40:20.217866 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:20.217882 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:20.218042 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:20.218518 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:20.218521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:20.218530 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 03:40:20.218643 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:20.218650 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.219556 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.219559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:20.220561 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:20.220573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.221107 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.221113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:20.221207 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:20.221240 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 03:40:20.234516 (+ 13276us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38163/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 03:40:20.234755 (+ 239us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.175.126:32971: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38163/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":145,"threads_started":1}
I20260810 03:40:20.235731 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 27736
2026-08-10T03:40:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (335 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 03:40:20.278510 22205 mini_oidc.cc:152] Starting JWKS server
I20260810 03:40:20.279443 22205 webserver.cc:460] Webserver started at https://127.0.0.1:39317/ using document root <none> and password file <none>
I20260810 03:40:20.279587 22205 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 03:40:20.279889 22205 webserver.cc:460] Webserver started at http://127.0.0.1:41547/ using document root <none> and password file <none>
2026-08-10T03:40:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:20Z Disabled control of system clock
I20260810 03:40:20.290918 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:33487
--webserver_interface=127.21.175.126
--webserver_port=0
--builtin_ntp_servers=127.21.175.84:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:33487
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39317/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 03:40:20.411564 27847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:20.411886 27847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:20.411929 27847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:20.416111 27847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:20.416193 27847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:20.416209 27847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:20.416226 27847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:20.416242 27847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 03:40:20.416296 27847 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 03:40:20.416328 27847 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39317/jwks.json
I20260810 03:40:20.421303 27847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:33487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:33487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39317/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:20.422600 27847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:20.424131 27847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:20.433558 27852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:20.433686 27853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:20.433734 27847 server_base.cc:1034] running on GCE node
W20260810 03:40:20.433591 27855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:20.434581 27847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:20.435791 27847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:20.436995 27847 hybrid_clock.cc:648] HybridClock initialized: now 1786333220436954 us; error 46 us; skew 500 ppm
I20260810 03:40:20.439167 27847 webserver.cc:460] Webserver started at http://127.21.175.126:44127/ using document root <none> and password file <none>
I20260810 03:40:20.439807 27847 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:20.439878 27847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:20.440109 27847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:20.442413 27847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "2ee271f604fc4add958f2768ed5d81c9"
format_stamp: "Formatted at 2026-08-10 03:40:20 on dist-test-slave-ss4n"
server_key: "7e0793ae23c68fc5c890b443641dc593"
server_key_iv: "aebfd2f862245a85affeb8f359670f71"
server_key_version: "encryptionkey@0"
I20260810 03:40:20.443043 27847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "2ee271f604fc4add958f2768ed5d81c9"
format_stamp: "Formatted at 2026-08-10 03:40:20 on dist-test-slave-ss4n"
server_key: "7e0793ae23c68fc5c890b443641dc593"
server_key_iv: "aebfd2f862245a85affeb8f359670f71"
server_key_version: "encryptionkey@0"
I20260810 03:40:20.447184 27847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:40:20.450065 27861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:20.451510 27847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:20.451690 27847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "2ee271f604fc4add958f2768ed5d81c9"
format_stamp: "Formatted at 2026-08-10 03:40:20 on dist-test-slave-ss4n"
server_key: "7e0793ae23c68fc5c890b443641dc593"
server_key_iv: "aebfd2f862245a85affeb8f359670f71"
server_key_version: "encryptionkey@0"
I20260810 03:40:20.451824 27847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:20.465322 27847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:20.466189 27847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:20.466425 27847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:20.474529 27847 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:33487
I20260810 03:40:20.474553 27919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:33487 every 8 connection(s)
I20260810 03:40:20.475704 27847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:20.478240 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 27847
I20260810 03:40:20.478375 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:40:20.478704 22205 external_mini_cluster.cc:1428] Setting key 542db98409eca5efe2ba9e694e37efb9
I20260810 03:40:20.481242 27920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:20.487340 27920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9: Bootstrap starting.
I20260810 03:40:20.489460 27924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:20.480197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.21.175.126:33487)
0810 03:40:20.481069 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:20.481089 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:20.481134 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:40:20.481985 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:20.482008 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:20.482548 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:20.482958 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:20.483473 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.484564 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.485649 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.486224 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.486417 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:20.487496 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:20.487504 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:20.487529 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:20.487969 (+ 440us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:20.488034 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:20.488058 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:20.488185 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:20.488361 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 03:40:20.488662 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":166,"threads_started":1}
I20260810 03:40:20.489889 27920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:20.490665 27920 log.cc:824] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:20.492895 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:40:20.493175 27920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9: No bootstrap required, opened a new log
I20260810 03:40:20.497179 27920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee271f604fc4add958f2768ed5d81c9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 33487 } }
I20260810 03:40:20.497422 27920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:20.497458 27920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ee271f604fc4add958f2768ed5d81c9, State: Initialized, Role: FOLLOWER
I20260810 03:40:20.497989 27920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee271f604fc4add958f2768ed5d81c9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 33487 } }
I20260810 03:40:20.498165 27920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:20.498258 27920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:20.498379 27920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:20.499473 27920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee271f604fc4add958f2768ed5d81c9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 33487 } }
I20260810 03:40:20.499841 27920 leader_election.cc:302] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ee271f604fc4add958f2768ed5d81c9; no voters:
I20260810 03:40:20.500229 27920 leader_election.cc:288] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:40:20.500391 27926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:20.500627 27926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [term 1 LEADER]: Becoming Leader. State: Replica: 2ee271f604fc4add958f2768ed5d81c9, State: Running, Role: LEADER
I20260810 03:40:20.500978 27926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee271f604fc4add958f2768ed5d81c9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 33487 } }
I20260810 03:40:20.501677 27920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:20.502887 27929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ee271f604fc4add958f2768ed5d81c9. Latest consensus state: current_term: 1 leader_uuid: "2ee271f604fc4add958f2768ed5d81c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee271f604fc4add958f2768ed5d81c9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 33487 } } }
I20260810 03:40:20.503016 27929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:20.503286 27927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ee271f604fc4add958f2768ed5d81c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee271f604fc4add958f2768ed5d81c9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 33487 } } }
I20260810 03:40:20.503379 27927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9 [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:20.503512 27936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:20.506937 27936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:20.513419 27936 catalog_manager.cc:1347] Generated new cluster ID: c409bb6353974d51bda44a65c1164410
I20260810 03:40:20.513556 27936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:20.536054 27936 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:40:20.537170 27936 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:20.552048 27936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ee271f604fc4add958f2768ed5d81c9: Generated new TSK 0
I20260810 03:40:20.552906 27936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:40:20.572773 27924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:20.567517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38110 (local address 127.21.175.126:33487)
0810 03:40:20.567679 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:20.567684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:20.567855 (+ 171us) server_negotiation.cc:407] Connection header received
0810 03:40:20.568071 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:20.568074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:20.568132 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:20.568225 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0810 03:40:20.568507 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.569040 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.570348 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.570565 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.570649 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:20.571022 (+ 373us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 03:40:20.572371 (+ 1349us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39317/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 03:40:20.572636 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38110: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39317/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 03:40:20.572870 27924 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38110: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39317/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 03:40:20.575676 27953 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:20.567425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:33487 (local address 127.0.0.1:38110)
0810 03:40:20.567754 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:20.567770 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:20.567894 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:20.568256 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:20.568259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:20.568268 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 03:40:20.568397 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:20.568403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.569182 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.569186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:20.570222 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:20.570235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.570702 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.570708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:20.570801 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:20.570833 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 03:40:20.572545 (+ 1712us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39317/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 03:40:20.575495 (+ 2950us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.175.126:33487: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39317/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":2687,"thread_start_us":119,"threads_started":1}
I20260810 03:40:20.576167 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 27847
E20260810 03:40:20.587610 27954 webserver.cc:533] Webserver: sslize failed:
E20260810 03:40:20.587705 27954 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T03:40:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (341 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T03:40:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:20Z Disabled control of system clock
I20260810 03:40:20.631570 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:37587
--webserver_interface=127.21.175.126
--webserver_port=0
--builtin_ntp_servers=127.21.175.84:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:37587
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/testchainca.cert with env {}
W20260810 03:40:20.753041 27958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:20.753517 27958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:20.753587 27958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:20.757876 27958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:20.757992 27958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:20.758023 27958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:20.758054 27958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:20.758095 27958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 03:40:20.758128 27958 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 03:40:20.758162 27958 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 03:40:20.763989 27958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:37587
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:37587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:20.765223 27958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:20.766381 27958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:40:20.775930 27958 server_base.cc:1034] running on GCE node
W20260810 03:40:20.775874 27966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:20.775874 27964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:20.775916 27963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:20.776696 27958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:20.777995 27958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:20.779167 27958 hybrid_clock.cc:648] HybridClock initialized: now 1786333220779147 us; error 35 us; skew 500 ppm
I20260810 03:40:20.789069 27958 webserver.cc:460] Webserver started at http://127.21.175.126:46487/ using document root <none> and password file <none>
I20260810 03:40:20.789845 27958 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:20.789924 27958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:20.790161 27958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:20.792462 27958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "c53f3fc597b94a6796af954606e02152"
format_stamp: "Formatted at 2026-08-10 03:40:20 on dist-test-slave-ss4n"
server_key: "426013bd5f6da642c74f012f54cd34e6"
server_key_iv: "322fa45fd975b81937b55e6d3db28d4f"
server_key_version: "encryptionkey@0"
I20260810 03:40:20.793097 27958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "c53f3fc597b94a6796af954606e02152"
format_stamp: "Formatted at 2026-08-10 03:40:20 on dist-test-slave-ss4n"
server_key: "426013bd5f6da642c74f012f54cd34e6"
server_key_iv: "322fa45fd975b81937b55e6d3db28d4f"
server_key_version: "encryptionkey@0"
I20260810 03:40:20.797382 27958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:40:20.800259 27972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:20.801827 27958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:20.802042 27958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "c53f3fc597b94a6796af954606e02152"
format_stamp: "Formatted at 2026-08-10 03:40:20 on dist-test-slave-ss4n"
server_key: "426013bd5f6da642c74f012f54cd34e6"
server_key_iv: "322fa45fd975b81937b55e6d3db28d4f"
server_key_version: "encryptionkey@0"
I20260810 03:40:20.802207 27958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:20.815793 27958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:20.816578 27958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:20.816809 27958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:20.824824 27958 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:37587
I20260810 03:40:20.824854 28030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:37587 every 8 connection(s)
I20260810 03:40:20.826004 27958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:20.829331 28031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:20.829619 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 27958
I20260810 03:40:20.829751 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:40:20.830122 22205 external_mini_cluster.cc:1428] Setting key 684a399775478c68ed652b057ee71ecc
I20260810 03:40:20.836932 28031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152: Bootstrap starting.
I20260810 03:40:20.839927 28031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:20.839938 28034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:20.831569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36352 (local address 127.21.175.126:37587)
0810 03:40:20.832301 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:20.832324 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:20.832376 (+ 52us) server_negotiation.cc:407] Connection header received
0810 03:40:20.833122 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:20.833143 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:20.833641 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:20.833998 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:20.834447 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.835421 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.836351 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.836864 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.837041 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:20.838062 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:20.838067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:20.838082 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:20.838498 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:20.838559 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:20.838582 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:20.838707 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:20.838873 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 03:40:20.839165 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":158,"threads_started":1}
I20260810 03:40:20.841022 28031 log.cc:824] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:20.843564 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:40:20.843909 28031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152: No bootstrap required, opened a new log
I20260810 03:40:20.848201 28031 raft_consensus.cc:348] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53f3fc597b94a6796af954606e02152" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37587 } }
I20260810 03:40:20.848487 28031 raft_consensus.cc:374] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:20.848527 28031 raft_consensus.cc:729] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c53f3fc597b94a6796af954606e02152, State: Initialized, Role: FOLLOWER
I20260810 03:40:20.849016 28031 consensus_queue.cc:260] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53f3fc597b94a6796af954606e02152" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37587 } }
I20260810 03:40:20.849179 28031 raft_consensus.cc:388] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:20.849284 28031 raft_consensus.cc:482] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:20.849416 28031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:20.850824 28031 raft_consensus.cc:504] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53f3fc597b94a6796af954606e02152" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37587 } }
I20260810 03:40:20.851219 28031 leader_election.cc:302] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c53f3fc597b94a6796af954606e02152; no voters:
I20260810 03:40:20.851629 28031 leader_election.cc:288] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:40:20.851992 28036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:20.852295 28036 raft_consensus.cc:686] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [term 1 LEADER]: Becoming Leader. State: Replica: c53f3fc597b94a6796af954606e02152, State: Running, Role: LEADER
I20260810 03:40:20.852702 28036 consensus_queue.cc:237] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53f3fc597b94a6796af954606e02152" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37587 } }
I20260810 03:40:20.853149 28031 sys_catalog.cc:564] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:20.854650 28037 sys_catalog.cc:455] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c53f3fc597b94a6796af954606e02152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53f3fc597b94a6796af954606e02152" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37587 } } }
I20260810 03:40:20.854823 28037 sys_catalog.cc:458] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:20.855058 28038 sys_catalog.cc:455] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c53f3fc597b94a6796af954606e02152. Latest consensus state: current_term: 1 leader_uuid: "c53f3fc597b94a6796af954606e02152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53f3fc597b94a6796af954606e02152" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37587 } } }
I20260810 03:40:20.855229 28038 sys_catalog.cc:458] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152 [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:20.855324 28048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:20.858706 28048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:20.866909 28048 catalog_manager.cc:1347] Generated new cluster ID: 2ca387fe1b844683babb095822f46759
I20260810 03:40:20.867030 28048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:20.901867 28048 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:40:20.902998 28048 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:20.915352 28048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c53f3fc597b94a6796af954606e02152: Generated new TSK 0
I20260810 03:40:20.916217 28048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:40:21.362957 28034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:20.932370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.21.175.126:37587)
0810 03:40:20.932540 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:20.932545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:20.932735 (+ 190us) server_negotiation.cc:407] Connection header received
0810 03:40:20.933050 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:20.933056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:20.933148 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:20.933293 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0810 03:40:20.933738 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.934368 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.935560 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.935789 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.935869 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:20.936322 (+ 453us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 03:40:21.362509 (+426187us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 03:40:21.362828 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36368: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 03:40:21.363083 28034 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36368: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 03:40:21.363216 28062 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:20.932291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:37587 (local address 127.0.0.1:36368)
0810 03:40:20.932638 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:20.932654 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:20.932823 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:20.933313 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:20.933318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:20.933327 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 03:40:20.933569 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:20.933593 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.934498 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.934502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:20.935438 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:20.935449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:20.935920 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:20.935924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:20.936030 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:20.936062 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 03:40:21.362793 (+426731us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 03:40:21.363008 (+ 215us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.175.126:37587: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":146,"threads_started":1}
I20260810 03:40:21.363837 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 27958
2026-08-10T03:40:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (765 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (132 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (132 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:40:21 dist-test-slave-ss4n krb5kdc[28075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:40:21 dist-test-slave-ss4n krb5kdc[28075](info): set up 2 sockets
Aug 10 03:40:21 dist-test-slave-ss4n krb5kdc[28075](info): commencing operation
krb5kdc: starting...
W20260810 03:40:22.762123 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:40:22 dist-test-slave-ss4n krb5kdc[28075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:40:22.905710 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:39065
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:39065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:23.024180 28089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:23.024482 28089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:23.024528 28089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:23.028753 28089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:23.028846 28089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:23.028867 28089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:23.028885 28089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:23.028900 28089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:40:23.034811 28089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:39065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:39065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:23.036001 28089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:23.037066 28089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:23.046403 28095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:23.046412 28094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:23.046461 28089 server_base.cc:1034] running on GCE node
W20260810 03:40:23.046424 28097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:23.047309 28089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:23.048563 28089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:23.049768 28089 hybrid_clock.cc:648] HybridClock initialized: now 1786333223049755 us; error 30 us; skew 500 ppm
Aug 10 03:40:23 dist-test-slave-ss4n krb5kdc[28075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333223, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:23.052842 28089 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:40:23.053964 28089 webserver.cc:460] Webserver started at http://127.21.175.126:36199/ using document root <none> and password file <none>
I20260810 03:40:23.054575 28089 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:23.054670 28089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:23.054922 28089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:23.057123 28089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "35372fb5e8a5452a94602f277d5916cd"
format_stamp: "Formatted at 2026-08-10 03:40:23 on dist-test-slave-ss4n"
server_key: "88be8148416bfabc6ac6b3553a0cc6f6"
server_key_iv: "98f42ca064f2e22a4da0e2138363d617"
server_key_version: "encryptionkey@0"
I20260810 03:40:23.057785 28089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "35372fb5e8a5452a94602f277d5916cd"
format_stamp: "Formatted at 2026-08-10 03:40:23 on dist-test-slave-ss4n"
server_key: "88be8148416bfabc6ac6b3553a0cc6f6"
server_key_iv: "98f42ca064f2e22a4da0e2138363d617"
server_key_version: "encryptionkey@0"
I20260810 03:40:23.061954 28089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:40:23.064692 28104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:23.066319 28089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:23.066586 28089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "35372fb5e8a5452a94602f277d5916cd"
format_stamp: "Formatted at 2026-08-10 03:40:23 on dist-test-slave-ss4n"
server_key: "88be8148416bfabc6ac6b3553a0cc6f6"
server_key_iv: "98f42ca064f2e22a4da0e2138363d617"
server_key_version: "encryptionkey@0"
I20260810 03:40:23.066747 28089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:23.083220 28089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:23.086723 28089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:23.086952 28089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:23.095515 28089 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:39065
I20260810 03:40:23.095531 28161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:39065 every 8 connection(s)
I20260810 03:40:23.096618 28089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:23.099629 28162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:23.103920 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 28089
I20260810 03:40:23.104036 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:40:23.104349 22205 external_mini_cluster.cc:1428] Setting key a294ab626b41d09640ec997f1026ecdc
I20260810 03:40:23.105901 28162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd: Bootstrap starting.
I20260810 03:40:23.108397 28162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:23.109213 28162 log.cc:824] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:23.111366 28162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd: No bootstrap required, opened a new log
Aug 10 03:40:23 dist-test-slave-ss4n krb5kdc[28075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:23.115629 28162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35372fb5e8a5452a94602f277d5916cd" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39065 } }
I20260810 03:40:23.115885 28162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:23.115988 28162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35372fb5e8a5452a94602f277d5916cd, State: Initialized, Role: FOLLOWER
I20260810 03:40:23.116506 28162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35372fb5e8a5452a94602f277d5916cd" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39065 } }
I20260810 03:40:23.116688 28162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:23.116767 28162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:23.116887 28162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:23.117996 28162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35372fb5e8a5452a94602f277d5916cd" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39065 } }
I20260810 03:40:23.118436 28162 leader_election.cc:302] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35372fb5e8a5452a94602f277d5916cd; no voters:
I20260810 03:40:23.118846 28162 leader_election.cc:288] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:40:23.118963 28167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:23.119248 28167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [term 1 LEADER]: Becoming Leader. State: Replica: 35372fb5e8a5452a94602f277d5916cd, State: Running, Role: LEADER
I20260810 03:40:23.119680 28167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35372fb5e8a5452a94602f277d5916cd" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39065 } }
I20260810 03:40:23.120517 28162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:23.122331 28165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:23.106028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39140 (local address 127.21.175.126:39065)
0810 03:40:23.106550 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:23.106567 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:23.106609 (+ 42us) server_negotiation.cc:407] Connection header received
0810 03:40:23.107357 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:23.107387 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:23.107798 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:23.108240 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:23.109212 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.110487 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.111473 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.112060 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.112266 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:23.115554 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:23.115560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:23.115578 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:23.115616 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:23.117843 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:23.118417 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:23.118422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:23.118428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:23.118720 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:23.119233 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:23.119240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:23.119242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:23.119913 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:23.120149 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:23.120485 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 03:40:23.120835 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":140,"threads_started":1}
I20260810 03:40:23.124148 28169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35372fb5e8a5452a94602f277d5916cd. Latest consensus state: current_term: 1 leader_uuid: "35372fb5e8a5452a94602f277d5916cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35372fb5e8a5452a94602f277d5916cd" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39065 } } }
I20260810 03:40:23.124275 28170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35372fb5e8a5452a94602f277d5916cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35372fb5e8a5452a94602f277d5916cd" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39065 } } }
I20260810 03:40:23.126663 28169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:23.126667 28170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd [sys.catalog]: This master's current role is: LEADER
W20260810 03:40:23.126487 28183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:40:23.127028 28183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:40:23.127152 28186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:23.127961 28186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:23.134438 28186 catalog_manager.cc:1347] Generated new cluster ID: 49eaaba02dab4416a45bee1277ff3e0a
I20260810 03:40:23.134536 28186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:23.154225 28186 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:40:23.155292 28186 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:23.167639 28186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35372fb5e8a5452a94602f277d5916cd: Generated new TSK 0
I20260810 03:40:23.168771 28186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:40:23.213667 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39065
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:23.336549 28190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:23.336884 28190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:23.336931 28190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:23.341558 28190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:23.341651 28190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:23.341758 28190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:40:23.347499 28190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:23.348765 28190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:23.349974 28190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:23.360492 28198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:23.360630 28196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:23.360607 28190 server_base.cc:1034] running on GCE node
W20260810 03:40:23.360742 28195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:23.361279 28190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:23.362106 28190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:23.363346 28190 hybrid_clock.cc:648] HybridClock initialized: now 1786333223363329 us; error 31 us; skew 500 ppm
Aug 10 03:40:23 dist-test-slave-ss4n krb5kdc[28075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333223, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:23.366760 28190 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:40:23.367871 28190 webserver.cc:460] Webserver started at http://127.21.175.65:37563/ using document root <none> and password file <none>
I20260810 03:40:23.368537 28190 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:23.368651 28190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:23.368924 28190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:23.371484 28190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "f8104aa5028c40d3848a12d5bd8e6310"
format_stamp: "Formatted at 2026-08-10 03:40:23 on dist-test-slave-ss4n"
server_key: "b194bcf612607b4824c3421dadad878f"
server_key_iv: "f03db222350b1f6512e37daaf77ca524"
server_key_version: "encryptionkey@0"
I20260810 03:40:23.372162 28190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "f8104aa5028c40d3848a12d5bd8e6310"
format_stamp: "Formatted at 2026-08-10 03:40:23 on dist-test-slave-ss4n"
server_key: "b194bcf612607b4824c3421dadad878f"
server_key_iv: "f03db222350b1f6512e37daaf77ca524"
server_key_version: "encryptionkey@0"
I20260810 03:40:23.376273 28190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:40:23.379148 28206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:23.380627 28190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:40:23.380811 28190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "f8104aa5028c40d3848a12d5bd8e6310"
format_stamp: "Formatted at 2026-08-10 03:40:23 on dist-test-slave-ss4n"
server_key: "b194bcf612607b4824c3421dadad878f"
server_key_iv: "f03db222350b1f6512e37daaf77ca524"
server_key_version: "encryptionkey@0"
I20260810 03:40:23.380959 28190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:23.404676 28190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:23.408613 28190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:23.408888 28190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:23.409669 28190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:23.410624 28190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:23.410703 28190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:23.410790 28190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:23.410851 28190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:23.421094 28190 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:43041
I20260810 03:40:23.421147 28327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:43041 every 8 connection(s)
I20260810 03:40:23.422369 28190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:40:23.432651 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 28190
I20260810 03:40:23.432783 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:40:23.433107 22205 external_mini_cluster.cc:1428] Setting key 9bbe96dc384a51620ee968378787ada5
Aug 10 03:40:23 dist-test-slave-ss4n krb5kdc[28075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333223, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:23.439451 28165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:23.424549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:52491 (local address 127.21.175.126:39065)
0810 03:40:23.424725 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:23.424731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:23.425903 (+ 1172us) server_negotiation.cc:407] Connection header received
0810 03:40:23.427425 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:23.427428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:23.427486 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:23.427577 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:23.428993 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.429713 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.430973 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.431359 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.431453 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:23.434696 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:23.434700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:23.434704 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:23.434735 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:23.436413 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:23.437220 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:23.437227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:23.437231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:23.437346 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:23.437885 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:23.437890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:23.437893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:23.438204 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:23.438321 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:23.438990 (+ 669us) server_negotiation.cc:299] Negotiation successful
0810 03:40:23.439173 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:40:23.440193 28330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:23.425138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:39065 (local address 127.21.175.65:52491)
0810 03:40:23.425721 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:23.425772 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:23.427156 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:23.427806 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:23.427814 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:23.428244 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:23.428818 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:23.428834 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.429839 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.429842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:23.430844 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:23.430853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.431510 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.431513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:23.431805 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:23.432454 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:23.432473 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:23.434432 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:23.436570 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:23.436581 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:23.436607 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:23.437066 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:23.437463 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:23.437466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:23.437467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:23.437770 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:23.438322 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:23.438328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:23.438687 (+ 359us) client_negotiation.cc:769] Sending connection context
0810 03:40:23.438871 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 03:40:23.439183 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":138,"threads_started":1}
I20260810 03:40:23.441895 28328 heartbeater.cc:347] Connected to a master server at 127.21.175.126:39065
I20260810 03:40:23.442274 28328 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:23.442981 28328 heartbeater.cc:511] Master 127.21.175.126:39065 requested a full tablet report, sending...
I20260810 03:40:23.444798 28122 ts_manager.cc:194] Registered new tserver with Master: f8104aa5028c40d3848a12d5bd8e6310 (127.21.175.65:43041)
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
I20260810 03:40:23.446262 28122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:52491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:40:23.469206 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39065
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:23.593221 28333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:23.593613 28333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:23.593704 28333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:23.597865 28333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:23.597983 28333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:23.598086 28333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:40:23.603591 28333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:23.604771 28333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:23.605888 28333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:23.616531 28341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:23.616580 28339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:23.616510 28338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:23.616779 28333 server_base.cc:1034] running on GCE node
I20260810 03:40:23.617311 28333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:23.618155 28333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:23.619347 28333 hybrid_clock.cc:648] HybridClock initialized: now 1786333223619327 us; error 39 us; skew 500 ppm
Aug 10 03:40:23 dist-test-slave-ss4n krb5kdc[28075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333223, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:23.622817 28333 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:40:23.623844 28333 webserver.cc:460] Webserver started at http://127.21.175.66:38839/ using document root <none> and password file <none>
I20260810 03:40:23.624406 28333 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:23.624468 28333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:23.624693 28333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:23.626947 28333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "a653e0bb16bb4bda9b9592b811d1106c"
format_stamp: "Formatted at 2026-08-10 03:40:23 on dist-test-slave-ss4n"
server_key: "221a9c4c20a606ae007e75dd2e1e63b9"
server_key_iv: "496f12c85741493475c6050ad53cf024"
server_key_version: "encryptionkey@0"
I20260810 03:40:23.627612 28333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "a653e0bb16bb4bda9b9592b811d1106c"
format_stamp: "Formatted at 2026-08-10 03:40:23 on dist-test-slave-ss4n"
server_key: "221a9c4c20a606ae007e75dd2e1e63b9"
server_key_iv: "496f12c85741493475c6050ad53cf024"
server_key_version: "encryptionkey@0"
I20260810 03:40:23.631610 28333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:40:23.634341 28349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:23.635929 28333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:23.636044 28333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "a653e0bb16bb4bda9b9592b811d1106c"
format_stamp: "Formatted at 2026-08-10 03:40:23 on dist-test-slave-ss4n"
server_key: "221a9c4c20a606ae007e75dd2e1e63b9"
server_key_iv: "496f12c85741493475c6050ad53cf024"
server_key_version: "encryptionkey@0"
I20260810 03:40:23.636142 28333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:23.649289 28333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:23.652837 28333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:23.653020 28333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:23.653694 28333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:23.654583 28333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:23.654635 28333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:23.654716 28333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:23.654759 28333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:23.664775 28333 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:46555
I20260810 03:40:23.664798 28467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:46555 every 8 connection(s)
I20260810 03:40:23.665889 28333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:40:23.668279 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 28333
I20260810 03:40:23.668422 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:40:23.668745 22205 external_mini_cluster.cc:1428] Setting key 0830b6660a8c2c842a545ff704344993
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
Aug 10 03:40:23 dist-test-slave-ss4n krb5kdc[28075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333223, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:23.688974 28165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:23.668007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:42885 (local address 127.21.175.126:39065)
0810 03:40:23.668153 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:23.668156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:23.669505 (+ 1349us) server_negotiation.cc:407] Connection header received
0810 03:40:23.671365 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:23.671368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:23.671419 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:23.671495 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:23.673125 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.673845 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.677002 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.677423 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.677579 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:23.684159 (+ 6580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:23.684162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:23.684164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:23.684195 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:23.686269 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:23.686960 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:23.686966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:23.686971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:23.687109 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:23.687532 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:23.687538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:23.687542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:23.687842 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:23.687950 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:23.688585 (+ 635us) server_negotiation.cc:299] Negotiation successful
0810 03:40:23.688758 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:40:23.690040 28470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:23.668640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:39065 (local address 127.21.175.66:42885)
0810 03:40:23.669266 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:23.669329 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:23.671111 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:23.671681 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:23.671688 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:23.672173 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:23.672925 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:23.672949 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.675570 (+ 2621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.675574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:23.676829 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:23.676843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.677673 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.677677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:23.677858 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:23.678600 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:23.678619 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:23.683897 (+ 5278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:23.686434 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:23.686441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:23.686459 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:23.686835 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:23.687241 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:23.687243 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:23.687245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:23.687420 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:23.687957 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:23.687964 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:23.688261 (+ 297us) client_negotiation.cc:769] Sending connection context
0810 03:40:23.688505 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 03:40:23.688871 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":185,"threads_started":1}
I20260810 03:40:23.691525 28468 heartbeater.cc:347] Connected to a master server at 127.21.175.126:39065
I20260810 03:40:23.691831 28468 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:23.692574 28468 heartbeater.cc:511] Master 127.21.175.126:39065 requested a full tablet report, sending...
I20260810 03:40:23.693925 28122 ts_manager.cc:194] Registered new tserver with Master: a653e0bb16bb4bda9b9592b811d1106c (127.21.175.66:46555)
I20260810 03:40:23.694717 28122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:42885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:40:23.705180 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39065
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:23.827289 28474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:23.827625 28474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:23.827721 28474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:23.831884 28474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:23.832000 28474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:23.832095 28474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:40:23.837767 28474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:23.838989 28474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:23.840205 28474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:40:23.850575 28474 server_base.cc:1034] running on GCE node
W20260810 03:40:23.850517 28482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:23.850543 28479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:23.850517 28480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:23.851246 28474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:23.851872 28474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:23.853044 28474 hybrid_clock.cc:648] HybridClock initialized: now 1786333223853023 us; error 33 us; skew 500 ppm
Aug 10 03:40:23 dist-test-slave-ss4n krb5kdc[28075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333223, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:23.856076 28474 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:40:23.857013 28474 webserver.cc:460] Webserver started at http://127.21.175.67:38797/ using document root <none> and password file <none>
I20260810 03:40:23.857683 28474 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:23.857754 28474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:23.857985 28474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:23.860162 28474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "e1607d0e67a04e76b97890683534e0ed"
format_stamp: "Formatted at 2026-08-10 03:40:23 on dist-test-slave-ss4n"
server_key: "ddad81d2f72cb116868cc66eabc87e0a"
server_key_iv: "963508d142c9fbfacae997c027db5d3a"
server_key_version: "encryptionkey@0"
I20260810 03:40:23.860786 28474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "e1607d0e67a04e76b97890683534e0ed"
format_stamp: "Formatted at 2026-08-10 03:40:23 on dist-test-slave-ss4n"
server_key: "ddad81d2f72cb116868cc66eabc87e0a"
server_key_iv: "963508d142c9fbfacae997c027db5d3a"
server_key_version: "encryptionkey@0"
I20260810 03:40:23.864970 28474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 03:40:23.867671 28489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:23.868961 28474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:40:23.869115 28474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "e1607d0e67a04e76b97890683534e0ed"
format_stamp: "Formatted at 2026-08-10 03:40:23 on dist-test-slave-ss4n"
server_key: "ddad81d2f72cb116868cc66eabc87e0a"
server_key_iv: "963508d142c9fbfacae997c027db5d3a"
server_key_version: "encryptionkey@0"
I20260810 03:40:23.869246 28474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:23.897678 28474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:23.901263 28474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:23.901578 28474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:23.902259 28474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:23.903162 28474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:23.903236 28474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:23.903311 28474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:23.903362 28474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:23.913074 28474 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:44689
I20260810 03:40:23.913094 28611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:44689 every 8 connection(s)
I20260810 03:40:23.914260 28474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:40:23.924080 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 28474
I20260810 03:40:23.924211 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:40:23.924512 22205 external_mini_cluster.cc:1428] Setting key f787abf8dd069b3caca6ec4481e25420
Aug 10 03:40:23 dist-test-slave-ss4n krb5kdc[28075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333223, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:23.933120 28165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:23.919111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:42945 (local address 127.21.175.126:39065)
0810 03:40:23.919960 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:23.919966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:23.920241 (+ 275us) server_negotiation.cc:407] Connection header received
0810 03:40:23.922204 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:23.922208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:23.922293 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:23.922410 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:23.923915 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.924594 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.926000 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.926294 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.926403 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:23.929456 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:23.929459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:23.929462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:23.929540 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:23.930880 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:23.931404 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:23.931407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:23.931408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:23.931493 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:23.931852 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:23.931855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:23.931856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:23.932105 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:23.932202 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:23.932854 (+ 652us) server_negotiation.cc:299] Negotiation successful
0810 03:40:23.932992 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663}
I20260810 03:40:23.933995 28614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:23.919480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:39065 (local address 127.21.175.67:42945)
0810 03:40:23.920020 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:23.920081 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:23.921930 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:23.922624 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:23.922632 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:23.923182 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:23.923712 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:23.923728 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.924721 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.924725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:23.925865 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:23.925878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.926473 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.926476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:23.926655 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:23.927492 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:23.927520 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:23.929280 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:23.931001 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:23.931007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:23.931022 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:23.931304 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:23.931581 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:23.931584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:23.931593 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:23.931736 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:23.932212 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:23.932218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:23.932571 (+ 353us) client_negotiation.cc:769] Sending connection context
0810 03:40:23.932787 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 03:40:23.933068 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":168,"threads_started":1}
I20260810 03:40:23.935283 28612 heartbeater.cc:347] Connected to a master server at 127.21.175.126:39065
I20260810 03:40:23.935585 28612 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:23.936189 28612 heartbeater.cc:511] Master 127.21.175.126:39065 requested a full tablet report, sending...
I20260810 03:40:23.937343 28122 ts_manager.cc:194] Registered new tserver with Master: e1607d0e67a04e76b97890683534e0ed (127.21.175.67:44689)
I20260810 03:40:23.937999 28122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:42945
I20260810 03:40:23.939615 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/test-admin.keytab.
Aug 10 03:40:23 dist-test-slave-ss4n krb5kdc[28075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:23.959678 22205 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 03:40:23.962694 28619 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:23.961198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.21.175.65:43041)
0810 03:40:23.961566 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:23.961572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:23.961598 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:40:23.961650 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:23.961655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:23.961841 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:23.961964 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:23.962440 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38530: BlockingRecv error: recv got EOF from 127.0.0.1:38530 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":111,"threads_started":1}
W20260810 03:40:23.962766 22205 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/krb5cc' of type FILE
Aug 10 03:40:23 dist-test-slave-ss4n krb5kdc[28075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:23.963845 22205 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 03:40:23 dist-test-slave-ss4n krb5kdc[28075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:23.964821 22205 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 03:40:23 dist-test-slave-ss4n krb5kdc[28075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.65@KRBTEST.COM
I20260810 03:40:23.975862 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:23.965794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38546 (local address 127.21.175.65:43041)
0810 03:40:23.965908 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:23.965911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:23.965992 (+ 81us) server_negotiation.cc:407] Connection header received
0810 03:40:23.966133 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:23.966135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:23.966177 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:23.966247 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:23.966859 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.967640 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.968691 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.969091 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.969185 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:23.971838 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:23.971848 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:23.971864 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:23.971891 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:23.973722 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:23.974287 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:23.974293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:23.974295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:23.974386 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:23.974722 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:23.974727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:23.974729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:23.975036 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:23.975296 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:23.975526 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 03:40:23.975703 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 03:40:23.980185 28619 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:23.978886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.21.175.65:43041)
0810 03:40:23.979011 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:23.979014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:23.979202 (+ 188us) server_negotiation.cc:407] Connection header received
0810 03:40:23.979312 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:23.979314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:23.979350 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:23.979414 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:23.980011 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38560: BlockingRecv error: recv got EOF from 127.0.0.1:38560 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 03:40:23.980369 22205 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/krb5cc' of type FILE
Aug 10 03:40:23 dist-test-slave-ss4n krb5kdc[28075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:23.981560 22205 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 03:40:23 dist-test-slave-ss4n krb5kdc[28075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:23.982539 22205 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 03:40:23 dist-test-slave-ss4n krb5kdc[28075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.65@KRBTEST.COM
I20260810 03:40:23.992600 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:23.983513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.21.175.65:43041)
0810 03:40:23.983648 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:23.983653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:23.983779 (+ 126us) server_negotiation.cc:407] Connection header received
0810 03:40:23.983932 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:23.983936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:23.983982 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:23.984064 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:23.984560 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.985198 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.986083 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:23.986337 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:23.986445 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:23.989215 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:23.989219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:23.989221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:23.989248 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:23.990835 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:23.991317 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:23.991321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:23.991322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:23.991401 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:23.991719 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:23.991724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:23.991725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:23.991918 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:23.992084 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:23.992317 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 03:40:23.992437 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 03:40:23.995934 28619 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:23.994781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38588 (local address 127.21.175.65:43041)
0810 03:40:23.994929 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:23.994933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:23.995096 (+ 163us) server_negotiation.cc:407] Connection header received
0810 03:40:23.995179 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:23.995182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:23.995219 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:23.995284 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:23.995822 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38588: BlockingRecv error: recv got EOF from 127.0.0.1:38588 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 03:40:23.996080 22205 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333190020696-22205-0/krb5kdc/krb5cc' of type FILE
Aug 10 03:40:23 dist-test-slave-ss4n krb5kdc[28075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:23.997215 22205 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 03:40:23 dist-test-slave-ss4n krb5kdc[28075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:23.998356 22205 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 03:40:24 dist-test-slave-ss4n krb5kdc[28075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.65@KRBTEST.COM
I20260810 03:40:24.007036 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:23.999282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38590 (local address 127.21.175.65:43041)
0810 03:40:23.999436 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:23.999440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:23.999516 (+ 76us) server_negotiation.cc:407] Connection header received
0810 03:40:23.999663 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:23.999665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:23.999703 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:23.999769 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:24.000261 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:24.000751 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:24.001573 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:24.001779 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:24.001859 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:24.004179 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:24.004183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:24.004185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:24.004207 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:24.005563 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:24.005939 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:24.005941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:24.005942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:24.006015 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:24.006237 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:24.006240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:24.006241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:24.006396 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:24.006511 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:24.006732 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 03:40:24.006845 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 03:40:24.008141 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 28190
I20260810 03:40:24.017639 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 28333
I20260810 03:40:24.027319 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 28474
I20260810 03:40:24.035624 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 28089
2026-08-10T03:40:24Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2408 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:40:24 dist-test-slave-ss4n krb5kdc[28633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:40:24 dist-test-slave-ss4n krb5kdc[28633](info): set up 2 sockets
Aug 10 03:40:24 dist-test-slave-ss4n krb5kdc[28633](info): commencing operation
krb5kdc: starting...
W20260810 03:40:25.187400 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:40:25 dist-test-slave-ss4n krb5kdc[28633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:40:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.126.keytab.
Entry for principal oryx/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.126.keytab.
I20260810 03:40:25.330662 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:36825
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.126.keytab
--principal=oryx/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:36825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:25.449899 28648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:25.450214 28648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:25.450258 28648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:25.454582 28648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:25.454675 28648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:25.454695 28648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:25.454712 28648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:25.454731 28648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:40:25.460484 28648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:36825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.126.keytab
--principal=oryx/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:36825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:25.461768 28648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:25.462913 28648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:25.472003 28653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:25.472137 28648 server_base.cc:1034] running on GCE node
W20260810 03:40:25.472028 28654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:25.472034 28656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:25.473035 28648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:25.474267 28648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:25.475430 28648 hybrid_clock.cc:648] HybridClock initialized: now 1786333225475396 us; error 48 us; skew 500 ppm
Aug 10 03:40:25 dist-test-slave-ss4n krb5kdc[28633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333225, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:25.478566 28648 init.cc:377] Logged in from keytab as oryx/127.21.175.126@KRBTEST.COM (short username oryx)
I20260810 03:40:25.479575 28648 webserver.cc:460] Webserver started at http://127.21.175.126:44373/ using document root <none> and password file <none>
I20260810 03:40:25.480195 28648 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:25.480261 28648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:25.480489 28648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:25.482748 28648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca"
format_stamp: "Formatted at 2026-08-10 03:40:25 on dist-test-slave-ss4n"
server_key: "7e83ebca6b69deb0ed470cf11ad76122"
server_key_iv: "f45baade71f27e4c25e07e805aaf4a16"
server_key_version: "encryptionkey@0"
I20260810 03:40:25.483428 28648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca"
format_stamp: "Formatted at 2026-08-10 03:40:25 on dist-test-slave-ss4n"
server_key: "7e83ebca6b69deb0ed470cf11ad76122"
server_key_iv: "f45baade71f27e4c25e07e805aaf4a16"
server_key_version: "encryptionkey@0"
I20260810 03:40:25.487740 28648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:40:25.490578 28663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:25.492141 28648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:25.492319 28648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca"
format_stamp: "Formatted at 2026-08-10 03:40:25 on dist-test-slave-ss4n"
server_key: "7e83ebca6b69deb0ed470cf11ad76122"
server_key_iv: "f45baade71f27e4c25e07e805aaf4a16"
server_key_version: "encryptionkey@0"
I20260810 03:40:25.492467 28648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:25.516403 28648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:25.521319 28648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:25.521642 28648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:25.530368 28648 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:36825
I20260810 03:40:25.530766 28717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:36825 every 8 connection(s)
I20260810 03:40:25.531687 28648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:25.535358 28718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:25.539584 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 28648
I20260810 03:40:25.539860 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:40:25.540217 22205 external_mini_cluster.cc:1428] Setting key 54a9c1e04143f49ac76d26db30fd4b08
I20260810 03:40:25.542225 28718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Bootstrap starting.
I20260810 03:40:25.545308 28718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:25.546303 28718 log.cc:824] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:25.548719 28718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: No bootstrap required, opened a new log
Aug 10 03:40:25 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM
I20260810 03:40:25.553674 28718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } }
I20260810 03:40:25.553998 28718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:25.554051 28718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c0ef9dc48b4bb5a29c2ca3b4eb44ca, State: Initialized, Role: FOLLOWER
I20260810 03:40:25.554569 28718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } }
I20260810 03:40:25.554765 28718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:25.554838 28718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:25.555015 28718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:25.555996 28718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } }
I20260810 03:40:25.556403 28718 leader_election.cc:302] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69c0ef9dc48b4bb5a29c2ca3b4eb44ca; no voters:
I20260810 03:40:25.556813 28718 leader_election.cc:288] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:40:25.556910 28723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:25.557124 28723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [term 1 LEADER]: Becoming Leader. State: Replica: 69c0ef9dc48b4bb5a29c2ca3b4eb44ca, State: Running, Role: LEADER
I20260810 03:40:25.557525 28723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } }
I20260810 03:40:25.557871 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:25.541886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58542 (local address 127.21.175.126:36825)
0810 03:40:25.542402 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:25.542423 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:25.542469 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:40:25.543339 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:25.543367 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:25.543857 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:25.544264 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:25.545201 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:25.546802 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:25.547901 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:25.548586 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:25.548764 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:25.551740 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:25.551749 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:25.551773 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:25.551815 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:25.554071 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:25.554654 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:25.554660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:25.554666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:25.554825 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:25.555185 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:25.555190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:25.555192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:25.555699 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:25.555955 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:25.556284 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 03:40:25.556696 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":170,"threads_started":1}
I20260810 03:40:25.558233 28718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:25.559309 28725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69c0ef9dc48b4bb5a29c2ca3b4eb44ca. Latest consensus state: current_term: 1 leader_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } }
I20260810 03:40:25.559455 28725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:25.559847 28724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } }
I20260810 03:40:25.559933 28724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:25.560380 28732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:25.563582 28732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:25.569849 28732 catalog_manager.cc:1347] Generated new cluster ID: 523daf2d91334e3999eb5540a2aa6f8c
I20260810 03:40:25.570058 28732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:25.590484 28732 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:40:25.591899 28732 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:25.600600 28732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Generated new TSK 0
I20260810 03:40:25.601574 28732 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.65.keytab.
Entry for principal oryx/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.65.keytab.
I20260810 03:40:25.638265 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36825
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.65.keytab
--principal=oryx/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:25.763095 28747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:25.763458 28747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:25.763562 28747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:25.767979 28747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:25.768108 28747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:25.768221 28747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:40:25.773700 28747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.65.keytab
--principal=oryx/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:25.774881 28747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:25.775977 28747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:25.786187 28753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:25.786240 28752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:25.786330 28747 server_base.cc:1034] running on GCE node
W20260810 03:40:25.786187 28755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:25.787006 28747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:25.787712 28747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:25.788921 28747 hybrid_clock.cc:648] HybridClock initialized: now 1786333225788902 us; error 33 us; skew 500 ppm
Aug 10 03:40:25 dist-test-slave-ss4n krb5kdc[28633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333225, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:25.796044 28747 init.cc:377] Logged in from keytab as oryx/127.21.175.65@KRBTEST.COM (short username oryx)
I20260810 03:40:25.797091 28747 webserver.cc:460] Webserver started at http://127.21.175.65:34813/ using document root <none> and password file <none>
I20260810 03:40:25.797801 28747 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:25.797926 28747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:25.798187 28747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:25.800376 28747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "1061eeaa37a74bac8059dae22ab9da04"
format_stamp: "Formatted at 2026-08-10 03:40:25 on dist-test-slave-ss4n"
server_key: "52af0c2de9e50bc2dffd07fe13b93801"
server_key_iv: "872a744622c4848e6e1138aae6a2a585"
server_key_version: "encryptionkey@0"
I20260810 03:40:25.800993 28747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "1061eeaa37a74bac8059dae22ab9da04"
format_stamp: "Formatted at 2026-08-10 03:40:25 on dist-test-slave-ss4n"
server_key: "52af0c2de9e50bc2dffd07fe13b93801"
server_key_iv: "872a744622c4848e6e1138aae6a2a585"
server_key_version: "encryptionkey@0"
I20260810 03:40:25.805119 28747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:40:25.807880 28762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:25.809288 28747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:25.809443 28747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "1061eeaa37a74bac8059dae22ab9da04"
format_stamp: "Formatted at 2026-08-10 03:40:25 on dist-test-slave-ss4n"
server_key: "52af0c2de9e50bc2dffd07fe13b93801"
server_key_iv: "872a744622c4848e6e1138aae6a2a585"
server_key_version: "encryptionkey@0"
I20260810 03:40:25.809621 28747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:25.850296 28747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:25.853458 28747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:25.853716 28747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:25.854395 28747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:25.855296 28747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:25.855369 28747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:25.855443 28747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:25.855497 28747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:25.865386 28747 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:37347
I20260810 03:40:25.865409 28883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:37347 every 8 connection(s)
I20260810 03:40:25.866365 28747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:40:25.867343 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 28747
I20260810 03:40:25.867462 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:40:25.867797 22205 external_mini_cluster.cc:1428] Setting key 78852607c3cf21e8f5d72dd43993122b
Aug 10 03:40:25 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333225, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.21.175.66@KRBTEST.COM; defaulting to no policy
I20260810 03:40:25.883649 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:25.868765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:33267 (local address 127.21.175.126:36825)
0810 03:40:25.868936 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:25.868941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:25.869913 (+ 972us) server_negotiation.cc:407] Connection header received
0810 03:40:25.872057 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:25.872063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:25.872136 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:25.872252 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:25.873455 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:25.874230 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:25.876182 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:25.876620 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:25.876758 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:25.879529 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:25.879537 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:25.879539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:25.879567 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:25.881128 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:25.881748 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:25.881751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:25.881753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:25.881844 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:25.882299 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:25.882302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:25.882304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:25.882578 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:25.882698 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:25.883256 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 03:40:25.883425 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.175.66@KRBTEST.COM" created.
I20260810 03:40:25.884675 28887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:25.869116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.65:33267)
0810 03:40:25.869731 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:25.869793 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:25.871748 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:25.872390 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:25.872397 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:25.872779 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:25.873294 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:25.873308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:25.874398 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:25.874405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:25.876039 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:25.876054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:25.876808 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:25.876813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:25.877037 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:25.877778 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:25.877796 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:25.879333 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:25.881259 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:25.881265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:25.881283 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:25.881636 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:25.882011 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:25.882015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:25.882017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:25.882186 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:25.882699 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:25.882707 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:25.882994 (+ 287us) client_negotiation.cc:769] Sending connection context
0810 03:40:25.883200 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 03:40:25.883568 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":176,"threads_started":1}
I20260810 03:40:25.886350 28884 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36825
I20260810 03:40:25.886719 28884 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:25.887440 28884 heartbeater.cc:511] Master 127.21.175.126:36825 requested a full tablet report, sending...
I20260810 03:40:25.889451 28680 ts_manager.cc:194] Registered new tserver with Master: 1061eeaa37a74bac8059dae22ab9da04 (127.21.175.65:37347)
I20260810 03:40:25.891484 28680 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:33267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.66.keytab.
Entry for principal oryx/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.66.keytab.
I20260810 03:40:25.906390 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36825
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.66.keytab
--principal=oryx/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:26.029172 28890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:26.029543 28890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:26.029614 28890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:26.033875 28890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:26.033941 28890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:26.034004 28890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:40:26.039527 28890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.66.keytab
--principal=oryx/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:26.040659 28890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:26.041764 28890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:26.052030 28898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:26.052052 28900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:26.052196 28897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:26.052479 28890 server_base.cc:1034] running on GCE node
I20260810 03:40:26.052939 28890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:26.053764 28890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:26.054993 28890 hybrid_clock.cc:648] HybridClock initialized: now 1786333226054975 us; error 27 us; skew 500 ppm
Aug 10 03:40:26 dist-test-slave-ss4n krb5kdc[28633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333226, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:26.058323 28890 init.cc:377] Logged in from keytab as oryx/127.21.175.66@KRBTEST.COM (short username oryx)
I20260810 03:40:26.059549 28890 webserver.cc:460] Webserver started at http://127.21.175.66:36351/ using document root <none> and password file <none>
I20260810 03:40:26.060258 28890 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:26.060364 28890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:26.060658 28890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:26.063175 28890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "4e60af6f969edc9889d5053672224866"
server_key_iv: "b957a64194ebaa9796525b3c5f5cebcd"
server_key_version: "encryptionkey@0"
I20260810 03:40:26.063854 28890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "4e60af6f969edc9889d5053672224866"
server_key_iv: "b957a64194ebaa9796525b3c5f5cebcd"
server_key_version: "encryptionkey@0"
I20260810 03:40:26.068176 28890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:40:26.070968 28907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:26.072659 28890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:40:26.072835 28890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "4e60af6f969edc9889d5053672224866"
server_key_iv: "b957a64194ebaa9796525b3c5f5cebcd"
server_key_version: "encryptionkey@0"
I20260810 03:40:26.072979 28890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:26.096853 28890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:26.100634 28890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:26.100875 28890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:26.101655 28890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:26.102747 28890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:26.102828 28890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:26.102911 28890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:26.102974 28890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:26.113969 28890 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:34913
I20260810 03:40:26.114048 29025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:34913 every 8 connection(s)
I20260810 03:40:26.115155 28890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:40:26.125257 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 28890
I20260810 03:40:26.125425 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
Aug 10 03:40:26 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333226, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM
I20260810 03:40:26.125901 22205 external_mini_cluster.cc:1428] Setting key 644a8545bcb4f6b2a3ff2f1c5808624c
I20260810 03:40:26.131654 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.117322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:32775 (local address 127.21.175.126:36825)
0810 03:40:26.117468 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:26.117471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:26.118626 (+ 1155us) server_negotiation.cc:407] Connection header received
0810 03:40:26.119993 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:26.119997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:26.120050 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:26.120137 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:26.121414 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.122059 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.123216 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.123450 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.123535 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:26.126632 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:26.126635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:26.126638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:26.126700 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:26.128883 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.129542 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.129550 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.129555 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.129678 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.130088 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.130095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.130100 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.130432 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:26.130534 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:26.131327 (+ 793us) server_negotiation.cc:299] Negotiation successful
0810 03:40:26.131524 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:40:26.132588 29028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.117931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.66:32775)
0810 03:40:26.118444 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:26.118493 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:26.119743 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:26.120339 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:26.120347 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:26.120744 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:26.121255 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:26.121268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.122188 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.122191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:26.123084 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:26.123093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.123593 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.123596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:26.123741 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:26.124316 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:26.124332 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:26.126375 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:26.129040 (+ 2665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:26.129046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:26.129062 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:26.129377 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:26.129812 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:26.129814 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:26.129816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:26.129978 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:26.130558 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:26.130566 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:26.130998 (+ 432us) client_negotiation.cc:769] Sending connection context
0810 03:40:26.131256 (+ 258us) client_negotiation.cc:240] Negotiation successful
0810 03:40:26.131670 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":145,"threads_started":1}
I20260810 03:40:26.133922 29026 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36825
I20260810 03:40:26.134231 29026 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:26.134917 29026 heartbeater.cc:511] Master 127.21.175.126:36825 requested a full tablet report, sending...
I20260810 03:40:26.136246 28680 ts_manager.cc:194] Registered new tserver with Master: 0dda5f21ab0d4dd4bdef3d73df6bd04c (127.21.175.66:34913)
I20260810 03:40:26.137120 28680 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:32775
WARNING: no policy specified for oryx/127.21.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.67.keytab.
Entry for principal oryx/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.67.keytab.
I20260810 03:40:26.159849 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36825
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.67.keytab
--principal=oryx/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:26.277616 29031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:26.277921 29031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:26.277973 29031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:26.282215 29031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:26.282295 29031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:26.282356 29031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:40:26.288009 29031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.67.keytab
--principal=oryx/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:26.289108 29031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:26.290222 29031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:26.301123 29036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:26.301105 29041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:26.301198 29031 server_base.cc:1034] running on GCE node
W20260810 03:40:26.301131 29037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:26.301923 29031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:26.302745 29031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:26.303967 29031 hybrid_clock.cc:648] HybridClock initialized: now 1786333226303914 us; error 67 us; skew 500 ppm
Aug 10 03:40:26 dist-test-slave-ss4n krb5kdc[28633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333226, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:26.307395 29031 init.cc:377] Logged in from keytab as oryx/127.21.175.67@KRBTEST.COM (short username oryx)
I20260810 03:40:26.308427 29031 webserver.cc:460] Webserver started at http://127.21.175.67:42489/ using document root <none> and password file <none>
I20260810 03:40:26.309080 29031 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:26.309165 29031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:26.309407 29031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:26.311936 29031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "9c144606813640a981f6128257243438"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "47d08e4df635be0e276e3927188173b5"
server_key_iv: "fb4bdf8f8a9bfa7f6781047cce20dfed"
server_key_version: "encryptionkey@0"
I20260810 03:40:26.312646 29031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "9c144606813640a981f6128257243438"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "47d08e4df635be0e276e3927188173b5"
server_key_iv: "fb4bdf8f8a9bfa7f6781047cce20dfed"
server_key_version: "encryptionkey@0"
I20260810 03:40:26.317158 29031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:40:26.320243 29049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:26.321938 29031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:26.322100 29031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "9c144606813640a981f6128257243438"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "47d08e4df635be0e276e3927188173b5"
server_key_iv: "fb4bdf8f8a9bfa7f6781047cce20dfed"
server_key_version: "encryptionkey@0"
I20260810 03:40:26.322253 29031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:26.338534 29031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:26.342442 29031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:26.342617 29031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:26.343189 29031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:26.344029 29031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:26.344081 29031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:26.344157 29031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:26.344182 29031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:26.353865 29031 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:42793
I20260810 03:40:26.353892 29165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:42793 every 8 connection(s)
I20260810 03:40:26.354871 29031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:40:26.358101 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 29031
I20260810 03:40:26.358266 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:40:26.358630 22205 external_mini_cluster.cc:1428] Setting key 6dfaa467dc1f94240d44130d32ab599f
Aug 10 03:40:26 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333226, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM
I20260810 03:40:26.371793 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.357093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:59567 (local address 127.21.175.126:36825)
0810 03:40:26.357261 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:26.357264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:26.358302 (+ 1038us) server_negotiation.cc:407] Connection header received
0810 03:40:26.360233 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:26.360236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:26.360291 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:26.360390 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:26.361957 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.362522 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.363722 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.363956 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.364089 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:26.367283 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:26.367290 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:26.367295 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:26.367336 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:26.369274 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.370069 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.370073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.370074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.370189 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.370642 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.370644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.370645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.370846 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:26.370924 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:26.371533 (+ 609us) server_negotiation.cc:299] Negotiation successful
0810 03:40:26.371663 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:40:26.373288 29168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.357591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.67:59567)
0810 03:40:26.358128 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:26.358185 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:26.359952 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:26.360602 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:26.360611 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:26.361112 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:26.361784 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:26.361811 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.362659 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.362663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:26.363584 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:26.363595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.364140 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.364145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:26.364333 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:26.365036 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:26.365054 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:26.367046 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:26.369410 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:26.369417 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:26.369434 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:26.369937 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:26.370348 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:26.370351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:26.370352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:26.370531 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:26.370966 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:26.370973 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:26.371283 (+ 310us) client_negotiation.cc:769] Sending connection context
0810 03:40:26.371459 (+ 176us) client_negotiation.cc:240] Negotiation successful
0810 03:40:26.371787 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":150,"threads_started":1}
I20260810 03:40:26.374562 29166 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36825
I20260810 03:40:26.374855 29166 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:26.375425 29166 heartbeater.cc:511] Master 127.21.175.126:36825 requested a full tablet report, sending...
I20260810 03:40:26.376605 28680 ts_manager.cc:194] Registered new tserver with Master: 9c144606813640a981f6128257243438 (127.21.175.67:42793)
I20260810 03:40:26.377308 28680 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:59567
I20260810 03:40:26.386130 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.175.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.175.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.175.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.125.keytab.
Entry for principal oryx/127.21.175.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.125.keytab.
I20260810 03:40:26.417194 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.125:43497
--webserver_interface=127.21.175.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.125.keytab
--principal=oryx/127.21.175.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:36825,127.21.175.125:43497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:26.540108 29171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:26.540416 29171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:26.540460 29171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:26.544907 29171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:26.544979 29171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:26.545001 29171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:26.545024 29171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:26.545039 29171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:40:26.550688 29171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:36825,127.21.175.125:43497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.125.keytab
--principal=oryx/127.21.175.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.125:43497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.175.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
W20260810 03:40:26.550983 29171 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.175.126:36825,127.21.175.125:43497'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 03:40:26.551997 29171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:26.553167 29171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:26.562645 29176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:26.562631 29177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:26.562631 29179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:26.563395 29171 server_base.cc:1034] running on GCE node
I20260810 03:40:26.563997 29171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:26.565269 29171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:26.566442 29171 hybrid_clock.cc:648] HybridClock initialized: now 1786333226566427 us; error 77 us; skew 500 ppm
Aug 10 03:40:26 dist-test-slave-ss4n krb5kdc[28633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333226, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:26.569746 29171 init.cc:377] Logged in from keytab as oryx/127.21.175.125@KRBTEST.COM (short username oryx)
I20260810 03:40:26.570791 29171 webserver.cc:460] Webserver started at http://127.21.175.125:40673/ using document root <none> and password file <none>
I20260810 03:40:26.571390 29171 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:26.571458 29171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:26.571696 29171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:26.573953 29171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/data/instance:
uuid: "e889ae8ffce6406d80d7ed13193faeb1"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "f248bf42c9f1b276e79b9751c315caef"
server_key_iv: "d20eef8f4c7a285bf52fcec427f72417"
server_key_version: "encryptionkey@0"
I20260810 03:40:26.574574 29171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/wal/instance:
uuid: "e889ae8ffce6406d80d7ed13193faeb1"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "f248bf42c9f1b276e79b9751c315caef"
server_key_iv: "d20eef8f4c7a285bf52fcec427f72417"
server_key_version: "encryptionkey@0"
I20260810 03:40:26.578958 29171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:40:26.581655 29186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:26.583161 29171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:26.583318 29171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/wal
uuid: "e889ae8ffce6406d80d7ed13193faeb1"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "f248bf42c9f1b276e79b9751c315caef"
server_key_iv: "d20eef8f4c7a285bf52fcec427f72417"
server_key_version: "encryptionkey@0"
I20260810 03:40:26.583442 29171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:26.630541 29171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:26.634377 29171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:26.634632 29171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:26.642423 29171 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.125:43497
I20260810 03:40:26.642716 29243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.125:43497 every 8 connection(s)
I20260810 03:40:26.643586 29171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/data/info.pb
I20260810 03:40:26.646360 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 29171
I20260810 03:40:26.646881 29244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:26.651954 29244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:40:26 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.175.125@KRBTEST.COM
Aug 10 03:40:26 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333226, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.125@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM
I20260810 03:40:26.662863 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.648128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.21.175.125:43497)
0810 03:40:26.648655 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:26.648672 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:26.648710 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:40:26.649365 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:26.649382 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:26.649849 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:26.650204 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:26.651057 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.652059 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.653000 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.653533 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.653725 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:26.656591 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:26.656596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:26.656607 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:26.656638 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:26.658945 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.659657 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.659665 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.659672 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.659804 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.660222 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.660226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.660228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.660795 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:26.661013 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:26.661384 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 03:40:26.661772 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":166,"threads_started":1}
I20260810 03:40:26.664326 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.653989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58556 (local address 127.21.175.126:36825)
0810 03:40:26.654166 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:26.654172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:26.654556 (+ 384us) server_negotiation.cc:407] Connection header received
0810 03:40:26.654731 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:26.654742 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:26.654788 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:26.654872 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:26.655577 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.656254 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.657229 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.657534 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.657684 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:26.660426 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:26.660430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:26.660432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:26.660462 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:26.662315 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.662830 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.662833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.662835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.662936 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.663296 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.663300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.663302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.663588 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:26.663702 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:26.664116 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 03:40:26.664217 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:40:26.664435 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.653930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:58556)
0810 03:40:26.654432 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:26.654473 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:26.654620 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:26.654925 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:26.654931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:26.655292 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:26.655412 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:26.655418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.656402 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.656406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:26.657123 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:26.657133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.657673 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.657676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:26.657806 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:26.658262 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:26.658279 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:26.660308 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:26.662438 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:26.662447 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:26.662453 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:26.662717 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:26.663057 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:26.663062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:26.663066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:26.663182 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:26.663731 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:26.663736 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:26.663853 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 03:40:26.664047 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 03:40:26.664237 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":107,"threads_started":1}
I20260810 03:40:26.665957 29244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:40:26 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333226, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.125@KRBTEST.COM for oryx/127.21.175.125@KRBTEST.COM
I20260810 03:40:26.674679 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.666520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:43497 (local address 127.0.0.1:41718)
0810 03:40:26.666664 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:26.666680 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:26.666766 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:26.667086 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:26.667089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:26.667264 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:26.667351 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:26.667356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.668511 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.668514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:26.669039 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:26.669046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.669524 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.669527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:26.669696 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:26.670211 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:26.670224 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:26.671468 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:26.673132 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:26.673135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:26.673137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:26.673356 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:26.673695 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:26.673697 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:26.673699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:26.673783 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:26.674182 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:26.674185 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:26.674291 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 03:40:26.674397 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 03:40:26.674535 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":51}
I20260810 03:40:26.674727 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.666551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41718 (local address 127.21.175.125:43497)
0810 03:40:26.666665 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:26.666670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:26.666724 (+ 54us) server_negotiation.cc:407] Connection header received
0810 03:40:26.666897 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:26.666899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:26.666940 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:26.667077 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:26.667461 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.668384 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.669156 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.669388 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.669554 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:26.671632 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:26.671635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:26.671637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:26.671658 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:26.673018 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.673448 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.673450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.673452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.673582 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.673870 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.673872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.673873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.674051 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:26.674129 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:26.674447 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 03:40:26.674590 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":41}
I20260810 03:40:26.678321 29244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: Bootstrap starting.
I20260810 03:40:26.680845 29244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:26.681833 29244 log.cc:824] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:26.683873 29244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: No bootstrap required, opened a new log
I20260810 03:40:26.687606 29244 raft_consensus.cc:348] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } }
I20260810 03:40:26.687884 29244 raft_consensus.cc:374] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:26.687984 29244 raft_consensus.cc:729] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e889ae8ffce6406d80d7ed13193faeb1, State: Initialized, Role: FOLLOWER
I20260810 03:40:26.688445 29244 consensus_queue.cc:260] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } }
I20260810 03:40:26.688896 29252 sys_catalog.cc:455] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } } }
I20260810 03:40:26.689033 29252 sys_catalog.cc:458] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:40:26.689728 29244 sys_catalog.cc:564] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 03:40:26.694556 29263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:40:26.694670 29263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:40:26.696712 29171 master_runner.cc:428] Detected that this master e889ae8ffce6406d80d7ed13193faeb1 is joining an existing cluster
I20260810 03:40:26.696794 29171 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.175.125:43497
I20260810 03:40:26.707629 29269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.699084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:43497 (local address 127.0.0.1:41720)
0810 03:40:26.699375 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:26.699391 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:26.699500 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:26.699893 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:26.699898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:26.700148 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:26.700285 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:26.700293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.701216 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.701220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:26.701960 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:26.701969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.702622 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.702627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:26.702719 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:26.703339 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:26.703353 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:26.703875 (+ 522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:26.706032 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:26.706035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:26.706037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:26.706270 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:26.706528 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:26.706531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:26.706533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:26.706599 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:26.707034 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:26.707038 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:26.707128 (+ 90us) client_negotiation.cc:769] Sending connection context
0810 03:40:26.707240 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 03:40:26.707416 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":123,"threads_started":1}
I20260810 03:40:26.707659 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.699167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.21.175.125:43497)
0810 03:40:26.699304 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:26.699309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:26.699687 (+ 378us) server_negotiation.cc:407] Connection header received
0810 03:40:26.699731 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:26.699734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:26.699785 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:26.699872 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:26.700432 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.701102 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.702082 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.702482 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.702590 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:26.704021 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:26.704025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:26.704028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:26.704057 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:26.705918 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.706365 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.706369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.706371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.706439 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.706689 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.706694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.706699 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.706945 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:26.707034 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:26.707315 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 03:40:26.707481 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":62}
I20260810 03:40:26.708452 29270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.699553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:58564)
0810 03:40:26.699892 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:26.699905 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:26.700026 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:26.700317 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:26.700320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:26.700568 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:26.700691 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:26.700697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.701798 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.701801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:26.702520 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:26.702529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.703311 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.703314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:26.703408 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:26.703871 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:26.703885 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:26.704796 (+ 911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:26.706605 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:26.706612 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:26.706614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:26.706857 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:26.707159 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:26.707162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:26.707164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:26.707253 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:26.707776 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:26.707780 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:26.708038 (+ 258us) client_negotiation.cc:769] Sending connection context
0810 03:40:26.708148 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 03:40:26.708281 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":35,"spinlock_wait_cycles":2944,"thread_start_us":102,"threads_started":1}
I20260810 03:40:26.708467 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.699613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58564 (local address 127.21.175.126:36825)
0810 03:40:26.699759 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:26.699764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:26.699969 (+ 205us) server_negotiation.cc:407] Connection header received
0810 03:40:26.700158 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:26.700160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:26.700202 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:26.700287 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:26.700849 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.701684 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.702681 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.703046 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.703108 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:26.704940 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:26.704944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:26.704946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:26.704980 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:26.706479 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.706968 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.706971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.706973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.707067 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.707365 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.707370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.707372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.707605 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:26.707687 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:26.708207 (+ 520us) server_negotiation.cc:299] Negotiation successful
0810 03:40:26.708323 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 03:40:26 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333225, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.126@KRBTEST.COM for oryx/127.21.175.125@KRBTEST.COM
I20260810 03:40:26.724874 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.713949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41724 (local address 127.21.175.125:43497)
0810 03:40:26.714081 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:26.714086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:26.714435 (+ 349us) server_negotiation.cc:407] Connection header received
0810 03:40:26.714721 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:26.714724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:26.714785 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:26.714876 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:26.715589 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.716280 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.717590 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.717936 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.718045 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:26.720962 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:26.720965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:26.720967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:26.720992 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:26.722709 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.723334 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.723338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.723339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.723428 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.723762 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.723764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.723765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.724013 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:26.724102 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:26.724609 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 03:40:26.724729 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 03:40:26.725072 29271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.713902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:43497 (local address 127.0.0.1:41724)
0810 03:40:26.714299 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:26.714344 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:26.714573 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:26.714918 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:26.714926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:26.715332 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:26.715463 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:26.715469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.716388 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.716393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:26.717430 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:26.717445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.718073 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.718080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:26.718243 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:26.718940 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:26.718963 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:26.720791 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:26.722843 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:26.722855 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:26.722864 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:26.723215 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:26.723531 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:26.723537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:26.723540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:26.723657 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:26.724140 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:26.724153 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:26.724330 (+ 177us) client_negotiation.cc:769] Sending connection context
0810 03:40:26.724525 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 03:40:26.724799 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":133,"threads_started":1}
I20260810 03:40:26.727035 28680 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.175.125:43497: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1"
member_type: NON_VOTER
last_known_addr {
host: "127.21.175.125"
port: 43497
}
attrs {
promote: true
}
}
dest_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca"
cas_config_opid_index: -1
WARNING: no policy specified for oryx/127.21.175.124@KRBTEST.COM; defaulting to no policy
I20260810 03:40:26.727977 28680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: NON_VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: true } }
I20260810 03:40:26.730051 29275 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e889ae8ffce6406d80d7ed13193faeb1 (127.21.175.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: NON_VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: true } } }
I20260810 03:40:26.730983 29215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.175.124@KRBTEST.COM" created.
I20260810 03:40:26.731875 29275 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.175.125:43497
I20260810 03:40:26.732108 29275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: NON_VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: true } } }
I20260810 03:40:26.732214 29275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:26.732574 29215 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 FOLLOWER]: Refusing update from remote peer 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 03:40:26.733414 29274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: NON_VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:26.733783 29171 master.cc:537] Master@127.21.175.125:43497 shutting down...
W20260810 03:40:26.735782 28665 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 03:40:26.737059 28665 consensus_peers.cc:487] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca -> Peer e889ae8ffce6406d80d7ed13193faeb1 (127.21.175.125:43497): Couldn't send request to peer e889ae8ffce6406d80d7ed13193faeb1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 03:40:26.739679 29171 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:26.740072 29171 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:26.740216 29171 tablet_replica.cc:330] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.175.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.124.keytab.
Entry for principal oryx/127.21.175.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.124.keytab.
I20260810 03:40:26.755239 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.124:44501
--webserver_interface=127.21.175.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.124.keytab
--principal=oryx/127.21.175.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:36825,127.21.175.125:43497,127.21.175.124:44501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:40:26.756145 29171 master.cc:552] Master@127.21.175.125:43497 shutdown complete.
I20260810 03:40:26.756209 29171 master_runner.cc:305] Clearing existing system tablet
I20260810 03:40:26.756497 29171 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:26.756560 29171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:26.758603 29171 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 03:40:26.759603 29281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:26.759884 29171 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:26.760099 29171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 03:40:26.760159 29171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/wal
uuid: "e889ae8ffce6406d80d7ed13193faeb1"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "f248bf42c9f1b276e79b9751c315caef"
server_key_iv: "d20eef8f4c7a285bf52fcec427f72417"
server_key_version: "encryptionkey@0"
I20260810 03:40:26.762372 29171 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:26.764364 29171 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 03:40:26.764487 29171 log.cc:1196] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 03:40:26.764750 29171 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: Deleting consensus metadata
I20260810 03:40:26.764945 29171 master_runner.cc:315] Copying system tablet from 127.21.175.126:36825
I20260810 03:40:26.766396 29171 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.175.126:36825
I20260810 03:40:26.774900 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.767154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.21.175.126:36825)
0810 03:40:26.767301 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:26.767305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:26.767491 (+ 186us) server_negotiation.cc:407] Connection header received
0810 03:40:26.767678 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:26.767684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:26.767736 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:26.767821 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:26.768513 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.769056 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.769998 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.770236 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.770348 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:26.771586 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:26.771589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:26.771592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:26.771627 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:26.773130 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.773642 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.773650 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.773651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.773728 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:26.774020 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:26.774023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:26.774025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:26.774219 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:26.774328 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:26.774606 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 03:40:26.774717 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:40:26.774912 29287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.767056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:58580)
0810 03:40:26.767379 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:26.767402 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:26.767554 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:26.767842 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:26.767845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:26.768084 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:26.768376 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:26.768383 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.769172 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.769177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:26.769870 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:26.769879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:26.770356 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:26.770360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:26.770457 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:26.771065 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:26.771078 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:26.771418 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:26.773258 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:26.773263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:26.773265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:26.773545 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:26.773821 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:26.773826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:26.773828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:26.773919 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:26.774317 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:26.774320 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:26.774429 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 03:40:26.774545 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 03:40:26.774715 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":90,"threads_started":1}
I20260810 03:40:26.775677 28702 tablet_copy_service.cc:140] P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e889ae8ffce6406d80d7ed13193faeb1 ({username='oryx', principal='oryx/127.21.175.125@KRBTEST.COM'} at 127.0.0.1:58580)
I20260810 03:40:26.775883 28702 tablet_copy_service.cc:161] P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e889ae8ffce6406d80d7ed13193faeb1 at {username='oryx', principal='oryx/127.21.175.125@KRBTEST.COM'} at 127.0.0.1:58580: session id = e889ae8ffce6406d80d7ed13193faeb1-00000000000000000000000000000000
I20260810 03:40:26.777765 28702 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 03:40:26.779666 29171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 03:40:26.781889 29171 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: tablet copy: Starting download of 0 data blocks...
I20260810 03:40:26.782130 29171 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: tablet copy: Starting download of 1 WAL segments...
I20260810 03:40:26.784061 29171 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 03:40:26.785794 28702 tablet_copy_service.cc:342] P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Request end of tablet copy session e889ae8ffce6406d80d7ed13193faeb1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.175.125@KRBTEST.COM'} at 127.0.0.1:58580
I20260810 03:40:26.785929 28702 tablet_copy_service.cc:433] P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: ending tablet copy session e889ae8ffce6406d80d7ed13193faeb1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e889ae8ffce6406d80d7ed13193faeb1
I20260810 03:40:26.793459 29171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:26.795625 29288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:26.795639 29291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:26.795876 29171 server_base.cc:1034] running on GCE node
W20260810 03:40:26.797631 29289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:26.797950 29171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:26.798295 29171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:26.799459 29171 hybrid_clock.cc:648] HybridClock initialized: now 1786333226799437 us; error 62 us; skew 500 ppm
Aug 10 03:40:26 dist-test-slave-ss4n krb5kdc[28633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333226, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:26.801079 29171 init.cc:377] Logged in from keytab as oryx/127.21.175.125@KRBTEST.COM (short username oryx)
I20260810 03:40:26.801753 29171 webserver.cc:460] Webserver started at http://127.21.175.125:34787/ using document root <none> and password file <none>
I20260810 03:40:26.801975 29171 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:26.802034 29171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:26.803234 29171 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 03:40:26.804286 29298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:26.804530 29171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 03:40:26.804605 29171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/wal
uuid: "e889ae8ffce6406d80d7ed13193faeb1"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "f248bf42c9f1b276e79b9751c315caef"
server_key_iv: "d20eef8f4c7a285bf52fcec427f72417"
server_key_version: "encryptionkey@0"
I20260810 03:40:26.804692 29171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:26.813930 29171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:26.816136 29171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:26.821189 29171 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.125:43497
I20260810 03:40:26.821201 29354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.125:43497 every 8 connection(s)
I20260810 03:40:26.821930 29171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-1/data/info.pb
I20260810 03:40:26.823702 29355 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:40:26.824560 29355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: Bootstrap starting.
I20260810 03:40:26.834163 29355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:40:26.834503 29355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: Bootstrap complete.
I20260810 03:40:26.835050 29355 raft_consensus.cc:348] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: NON_VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: true } }
I20260810 03:40:26.835201 29355 raft_consensus.cc:729] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e889ae8ffce6406d80d7ed13193faeb1, State: Initialized, Role: LEARNER
I20260810 03:40:26.835325 29355 consensus_queue.cc:260] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: NON_VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: true } }
I20260810 03:40:26.835696 29358 sys_catalog.cc:455] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: NON_VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: true } } }
I20260810 03:40:26.835866 29355 sys_catalog.cc:564] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:26.835866 29358 sys_catalog.cc:458] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: This master's current role is: LEARNER
I20260810 03:40:26.839035 29369 catalog_manager.cc:1259] Loaded cluster ID: 523daf2d91334e3999eb5540a2aa6f8c
I20260810 03:40:26.839193 29369 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: loading cluster ID for follower catalog manager: success
I20260810 03:40:26.841073 29369 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: acquiring CA information for follower catalog manager: success
I20260810 03:40:26.842451 29369 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:40:26.895226 28884 heartbeater.cc:503] Master 127.21.175.126:36825 was elected leader, sending a full tablet report...
W20260810 03:40:26.923110 29279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:26.923424 29279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:26.923473 29279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:26.927690 29279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:26.927762 29279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:26.927783 29279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:26.927798 29279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:26.927847 29279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:40:26.933468 29279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:36825,127.21.175.125:43497,127.21.175.124:44501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.124.keytab
--principal=oryx/127.21.175.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.124:44501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.175.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:26.934702 29279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:26.935890 29279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:26.945466 29374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:26.945468 29377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:26.945466 29375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:26.945859 29279 server_base.cc:1034] running on GCE node
I20260810 03:40:26.946663 29279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:26.947958 29279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:26.949162 29279 hybrid_clock.cc:648] HybridClock initialized: now 1786333226949143 us; error 33 us; skew 500 ppm
Aug 10 03:40:26 dist-test-slave-ss4n krb5kdc[28633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333226, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:26.952210 29279 init.cc:377] Logged in from keytab as oryx/127.21.175.124@KRBTEST.COM (short username oryx)
I20260810 03:40:26.953276 29279 webserver.cc:460] Webserver started at http://127.21.175.124:45645/ using document root <none> and password file <none>
I20260810 03:40:26.953925 29279 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:26.954053 29279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:26.954301 29279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:26.956429 29279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/data/instance:
uuid: "610504ba740c46c88004c6b8611b083e"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "6145d1e2cfc627ee0e5f82c7011abd7d"
server_key_iv: "b1409e568cb30463833bad0ec061fe59"
server_key_version: "encryptionkey@0"
I20260810 03:40:26.957055 29279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/wal/instance:
uuid: "610504ba740c46c88004c6b8611b083e"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "6145d1e2cfc627ee0e5f82c7011abd7d"
server_key_iv: "b1409e568cb30463833bad0ec061fe59"
server_key_version: "encryptionkey@0"
I20260810 03:40:26.961301 29279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 03:40:26.964020 29384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:26.965408 29279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:40:26.965621 29279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/wal
uuid: "610504ba740c46c88004c6b8611b083e"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "6145d1e2cfc627ee0e5f82c7011abd7d"
server_key_iv: "b1409e568cb30463833bad0ec061fe59"
server_key_version: "encryptionkey@0"
I20260810 03:40:26.965750 29279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:26.978683 29279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:26.981711 29279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:26.981926 29279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:26.989806 29279 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.124:44501
I20260810 03:40:26.990111 29441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.124:44501 every 8 connection(s)
I20260810 03:40:26.990934 29279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/data/info.pb
I20260810 03:40:26.993839 29442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:26.994990 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 29279
I20260810 03:40:26.998325 29442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333226, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.124@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.175.124@KRBTEST.COM
I20260810 03:40:27.011304 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.000154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58590 (local address 127.21.175.126:36825)
0810 03:40:27.000285 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.000289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.000657 (+ 368us) server_negotiation.cc:407] Connection header received
0810 03:40:27.000808 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.000810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.000853 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.000914 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.001889 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.002443 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.003464 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.003722 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.003837 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.006641 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.006652 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.006655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.006685 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.008671 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.009301 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.009308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.009313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.009423 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.009952 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.009957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.009961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.010248 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.010379 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.010940 (+ 561us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.011120 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:40:27.012168 29444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:26.996829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42782 (local address 127.21.175.124:44501)
0810 03:40:26.997341 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:26.997358 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:26.997398 (+ 40us) server_negotiation.cc:407] Connection header received
0810 03:40:26.998260 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:26.998280 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:26.998701 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:26.999174 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.000238 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.001915 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.002877 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.003430 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.003635 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.007066 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.007073 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.007085 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.007109 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.009260 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.009909 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.009919 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.009921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.010026 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.010317 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.010322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.010325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.010673 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.010879 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.011107 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.011312 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":354,"thread_start_us":156,"threads_started":1}
I20260810 03:40:27.012223 29445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.000042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:58590)
0810 03:40:27.000509 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.000563 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.000695 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.001014 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.001020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.001535 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:27.001750 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.001763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.002589 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.002592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.003335 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.003344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.003876 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.003880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.003989 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.004578 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:27.004594 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:27.006427 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:27.008791 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.008799 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.008808 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.009187 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.009609 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.009614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.009616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.009823 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.010377 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:27.010383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:27.010666 (+ 283us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.010867 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.011252 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":43,"thread_start_us":154,"threads_started":1}
I20260810 03:40:27.013599 29442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333226, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.124@KRBTEST.COM for oryx/127.21.175.125@KRBTEST.COM
I20260810 03:40:27.023310 29445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.014104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:43497 (local address 127.0.0.1:41738)
0810 03:40:27.014490 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.014504 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.014583 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.014996 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.014999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.015189 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:27.015312 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.015318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.016397 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.016400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.016970 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.016978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.017457 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.017459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.017652 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.018187 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:27.018199 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:27.019564 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:27.021672 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.021675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.021677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.021911 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.022252 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.022255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.022256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.022345 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.022869 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:27.022872 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:27.022966 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.023053 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.023178 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312}
I20260810 03:40:27.023418 29446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.014256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.21.175.125:43497)
0810 03:40:27.014783 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.014786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.014799 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:40:27.014836 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.014839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.014877 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.014953 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.015480 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.016271 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.017102 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.017347 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.017443 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.019758 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.019762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.019764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.019793 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.021412 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.022022 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.022025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.022027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.022125 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.022500 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.022504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.022505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.022757 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.022880 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.023134 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.023270 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":105,"threads_started":1}
I20260810 03:40:27.024371 29442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333226, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.124@KRBTEST.COM for oryx/127.21.175.124@KRBTEST.COM
I20260810 03:40:27.033712 29445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.025066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:44501 (local address 127.0.0.1:42788)
0810 03:40:27.025177 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.025192 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.025275 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.025759 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.025761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.025903 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:27.026014 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.026021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.027275 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.027277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.027800 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.027807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.028300 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.028302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.028386 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.028937 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:27.028948 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:27.030345 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:27.031976 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.031979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.031980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.032218 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.032514 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.032516 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.032518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.032610 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.033033 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:27.033035 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:27.033134 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.033242 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.033383 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":15}
I20260810 03:40:27.033712 29444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.025252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.21.175.124:44501)
0810 03:40:27.025426 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.025431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.025450 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:40:27.025536 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.025540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.025599 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.025726 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.026154 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.027164 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.027949 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.028198 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.028290 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.030487 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.030491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.030493 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.030513 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.031839 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.032318 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.032321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.032322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.032397 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.032703 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.032705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.032706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.032891 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.032979 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.033305 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.033435 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":66}
I20260810 03:40:27.036872 29442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: Bootstrap starting.
I20260810 03:40:27.039484 29442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:27.040403 29442 log.cc:824] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:27.042613 29442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: No bootstrap required, opened a new log
I20260810 03:40:27.045794 29442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } }
I20260810 03:40:27.046062 29442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:27.046109 29442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 610504ba740c46c88004c6b8611b083e, State: Initialized, Role: FOLLOWER
I20260810 03:40:27.046518 29442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } }
I20260810 03:40:27.046973 29448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } } }
I20260810 03:40:27.047150 29448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:40:27.047782 29442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [sys.catalog]: configured and running, proceeding with master startup.
W20260810 03:40:27.053069 29459 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:40:27.053194 29459 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:40:27.057037 29279 master_runner.cc:428] Detected that this master 610504ba740c46c88004c6b8611b083e is joining an existing cluster
I20260810 03:40:27.057119 29279 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.175.124:44501
I20260810 03:40:27.066562 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 28747
I20260810 03:40:27.082567 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:37347
--local_ip_for_outbound_sockets=127.21.175.65
--tserver_master_addrs=127.21.175.126:36825
--webserver_port=34813
--webserver_interface=127.21.175.65
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.65.keytab
--principal=oryx/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.175.126:36825,127.21.175.125:43497
--tserver_master_addrs=127.21.175.126:36825,127.21.175.125:43497,127.21.175.124:44501 with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:40:27.094735 29444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.060584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.21.175.124:44501)
0810 03:40:27.060931 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.060933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.061320 (+ 387us) server_negotiation.cc:407] Connection header received
0810 03:40:27.061449 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.061452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.061509 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.061558 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.062451 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.063222 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.064488 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.064794 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.064856 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.090507 (+ 25651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.090511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.090514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.090541 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.092346 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.092904 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.092910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.092913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.093029 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.093436 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.093442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.093446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.093796 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.093888 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.094329 (+ 441us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.094484 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289}
I20260810 03:40:27.095413 29466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.060521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:44501 (local address 127.0.0.1:42790)
0810 03:40:27.061244 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.061257 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.061365 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.061713 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.061717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.062055 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:27.062329 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.062337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.063363 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.063367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.064372 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.064382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.064985 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.064988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.065149 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.065769 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:27.065786 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:27.090296 (+ 24510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:27.092487 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.092490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.092492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.092769 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.093174 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.093179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.093184 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.093311 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.093978 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:27.093984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:27.094115 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.095019 (+ 904us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.095199 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":52,"spinlock_wait_cycles":6400,"thread_start_us":103,"threads_started":1}
I20260810 03:40:27.096935 29468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.060030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:58600)
0810 03:40:27.060783 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.060797 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.060942 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.061429 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.061441 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.061853 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:27.062021 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.062030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.063203 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.063206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.064176 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.064187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.065142 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.065146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.065247 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.090184 (+ 24937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:27.090225 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:27.090802 (+ 577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:27.094788 (+ 3986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.094790 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.094792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.095007 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.095822 (+ 815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.095826 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.095828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.095919 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.096397 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:27.096401 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:27.096492 (+ 91us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.096601 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.096759 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":1,"spinlock_wait_cycles":22912,"thread_start_us":93,"threads_started":1}
I20260810 03:40:27.097993 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.060080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.21.175.126:36825)
0810 03:40:27.060223 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.060228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.060876 (+ 648us) server_negotiation.cc:407] Connection header received
0810 03:40:27.061078 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.061081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.061167 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.061225 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.062199 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.063051 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.064324 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.064689 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.064776 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.091061 (+ 26285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.091067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.091073 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.091114 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.094473 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.095401 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.095405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.095407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.095503 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.096030 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.096033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.096035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.096262 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.096354 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.097714 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.097837 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:40:27.098695 29446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.061010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.21.175.125:43497)
0810 03:40:27.061146 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.061152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.061167 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:40:27.061210 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.061213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.061262 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.061339 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.062477 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.063263 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.064733 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.065090 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.065181 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.091815 (+ 26634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.091819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.091822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.091855 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.094210 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.097168 (+ 2958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.097172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.097174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.097282 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.097689 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.097693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.097695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.097920 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.098010 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.098358 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.098507 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:40:27.099234 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.059999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:43497 (local address 127.0.0.1:41746)
0810 03:40:27.060521 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.060535 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.060647 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.061466 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.061469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.062048 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:27.062330 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.062337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.063391 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.063396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.064615 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.064623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.065597 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.065600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.065708 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.066193 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:27.066213 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:27.091595 (+ 25382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:27.094598 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.094610 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.094612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.097035 (+ 2423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.097405 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.097407 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.097410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.097576 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.098084 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:27.098087 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:27.098175 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.098890 (+ 715us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.099052 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":81,"spinlock_wait_cycles":3712,"thread_start_us":123,"threads_started":1}
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333225, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.126@KRBTEST.COM for oryx/127.21.175.124@KRBTEST.COM
I20260810 03:40:27.109874 29448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:27.110083 29448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } }
I20260810 03:40:27.111339 29448 leader_election.cc:288] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69c0ef9dc48b4bb5a29c2ca3b4eb44ca (127.21.175.126:36825), e889ae8ffce6406d80d7ed13193faeb1 (127.21.175.125:43497)
I20260810 03:40:27.111702 28691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "610504ba740c46c88004c6b8611b083e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" is_pre_election: true
I20260810 03:40:27.111835 29327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "610504ba740c46c88004c6b8611b083e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e889ae8ffce6406d80d7ed13193faeb1" is_pre_election: true
I20260810 03:40:27.112030 29327 raft_consensus.cc:1799] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 LEARNER]: Handling vote request from an unknown peer 610504ba740c46c88004c6b8611b083e
I20260810 03:40:27.112164 29327 raft_consensus.cc:2396] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 610504ba740c46c88004c6b8611b083e for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260810 03:40:27.112530 29386 leader_election.cc:302] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610504ba740c46c88004c6b8611b083e; no voters: 69c0ef9dc48b4bb5a29c2ca3b4eb44ca, e889ae8ffce6406d80d7ed13193faeb1
I20260810 03:40:27.113005 29448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:27.114419 29448 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 03:40:27.116330 29444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.102433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.21.175.124:44501)
0810 03:40:27.102555 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.102559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.102573 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:40:27.102754 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.102759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.102828 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.102928 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.103400 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.104270 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.105519 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.105894 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.106011 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.108596 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.108600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.108614 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.108642 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.111069 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.114904 (+ 3835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.114909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.114910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.115013 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.115351 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.115355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.115357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.115595 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.115710 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.116039 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.116184 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:40:27.116437 29271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.102354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:44501 (local address 127.0.0.1:42798)
0810 03:40:27.102520 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.102534 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.102615 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.102963 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.102966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.103164 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:27.103273 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.103280 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.104420 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.104424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.105342 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.105354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.106022 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.106026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.106152 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.106722 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:27.106758 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:27.108430 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:27.111198 (+ 2768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.111202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.111204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.114782 (+ 3578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.115130 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.115133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.115135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.115244 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.115718 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:27.115721 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:27.115844 (+ 123us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.115968 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.116252 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260810 03:40:27.117993 28680 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.175.124:44501: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "610504ba740c46c88004c6b8611b083e"
member_type: NON_VOTER
last_known_addr {
host: "127.21.175.124"
port: 44501
}
attrs {
promote: true
}
}
dest_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca"
cas_config_opid_index: 5
I20260810 03:40:27.118649 28680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: NON_VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: true } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true } }
I20260810 03:40:27.119802 29275 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 610504ba740c46c88004c6b8611b083e (127.21.175.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: NON_VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: true } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true } } }
I20260810 03:40:27.120481 29414 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 1 FOLLOWER]: Refusing update from remote peer 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 03:40:27.121047 29274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:27.121390 29275 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.175.124:44501
I20260810 03:40:27.121883 29274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: NON_VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: true } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true } } }
I20260810 03:40:27.122015 29274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:27.122219 29279 master.cc:537] Master@127.21.175.124:44501 shutting down...
W20260810 03:40:27.124775 28665 consensus_peers.cc:487] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca -> Peer 610504ba740c46c88004c6b8611b083e (127.21.175.124:44501): Couldn't send request to peer 610504ba740c46c88004c6b8611b083e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 03:40:27.129873 29271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.120024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:43497 (local address 127.0.0.1:41760)
0810 03:40:27.120430 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.120446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.120552 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.125167 (+ 4615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.125171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.125195 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:27.125305 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.125311 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.126738 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.126741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.128048 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.128060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.129386 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.129390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.129517 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.129569 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.129644 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.129734 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322}
I20260810 03:40:27.130332 29446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.120294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41760 (local address 127.21.175.125:43497)
0810 03:40:27.124034 (+ 3740us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.124040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.124059 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:40:27.124119 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.124122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.124187 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.124289 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:27.125601 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.126401 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.128206 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.128902 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.129019 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.129170 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.130118 (+ 948us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.130205 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3624}
I20260810 03:40:27.130800 29327 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 LEARNER]: Refusing update from remote peer 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 03:40:27.132975 29470 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: NON_VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 03:40:27.133705 29279 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:27.134006 29279 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:27.134109 29279 tablet_replica.cc:330] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: stopping tablet replica
I20260810 03:40:27.135370 29327 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 610504ba740c46c88004c6b8611b083e (127.21.175.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: NON_VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: true } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true } } }
I20260810 03:40:27.137458 29358 sys_catalog.cc:455] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69c0ef9dc48b4bb5a29c2ca3b4eb44ca. Latest consensus state: current_term: 1 leader_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: NON_VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: true } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true } } }
I20260810 03:40:27.137652 29358 sys_catalog.cc:458] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: This master's current role is: LEARNER
I20260810 03:40:27.137732 29358 sys_catalog.cc:455] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: NON_VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: true } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true } } }
I20260810 03:40:27.137795 29358 sys_catalog.cc:458] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: This master's current role is: LEARNER
I20260810 03:40:27.137950 29274 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: attempting to promote NON_VOTER e889ae8ffce6406d80d7ed13193faeb1 to VOTER
I20260810 03:40:27.138440 29274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true } }
I20260810 03:40:27.139846 29026 heartbeater.cc:503] Master 127.21.175.126:36825 was elected leader, sending a full tablet report...
W20260810 03:40:27.143824 28665 consensus_peers.cc:487] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca -> Peer 610504ba740c46c88004c6b8611b083e (127.21.175.124:44501): Couldn't send request to peer 610504ba740c46c88004c6b8611b083e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 03:40:27.144572 29327 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 LEARNER]: Refusing update from remote peer 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 03:40:27.145123 29275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 03:40:27.148072 29327 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e889ae8ffce6406d80d7ed13193faeb1 (127.21.175.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true } } }
I20260810 03:40:27.150238 29472 sys_catalog.cc:455] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true } } }
I20260810 03:40:27.150370 29472 sys_catalog.cc:458] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:40:27.147423 29474 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e889ae8ffce6406d80d7ed13193faeb1 (127.21.175.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true } } }
I20260810 03:40:27.152729 29274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true } } }
I20260810 03:40:27.152854 29274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:27.153101 29274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true } } }
I20260810 03:40:27.153199 29274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:27.155866 29279 master.cc:552] Master@127.21.175.124:44501 shutdown complete.
I20260810 03:40:27.155927 29279 master_runner.cc:305] Clearing existing system tablet
I20260810 03:40:27.156200 29279 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:27.156262 29279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:27.157699 29279 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 03:40:27.158685 29476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:27.159154 29279 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:27.159550 29279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 03:40:27.159648 29279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/wal
uuid: "610504ba740c46c88004c6b8611b083e"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "6145d1e2cfc627ee0e5f82c7011abd7d"
server_key_iv: "b1409e568cb30463833bad0ec061fe59"
server_key_version: "encryptionkey@0"
I20260810 03:40:27.162240 29279 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:27.165096 29279 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 03:40:27.165302 29279 log.cc:1196] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 03:40:27.165735 29279 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: Deleting consensus metadata
I20260810 03:40:27.165975 29279 master_runner.cc:315] Copying system tablet from 127.21.175.126:36825
I20260810 03:40:27.167282 29279 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: tablet copy: Beginning tablet copy session from remote peer at address 127.21.175.126:36825
I20260810 03:40:27.179913 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.168095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58602 (local address 127.21.175.126:36825)
0810 03:40:27.168243 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.168248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.169918 (+ 1670us) server_negotiation.cc:407] Connection header received
0810 03:40:27.170068 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.170072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.170132 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.170225 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.170838 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.171655 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.172917 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.173259 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.173366 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.175441 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.175446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.175449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.175477 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.177663 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.178227 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.178231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.178233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.178333 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.178815 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.178819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.178821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.179070 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.179151 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.179585 (+ 434us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.179718 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:40:27.180832 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.167955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:58602)
0810 03:40:27.169782 (+ 1827us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.169799 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.169953 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.170287 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.170290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.170541 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:27.170700 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.170708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.171784 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.171788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.172781 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.172793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.173399 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.173403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.173550 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.174775 (+ 1225us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:27.174790 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:27.175267 (+ 477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:27.177803 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.177806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.177809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.178098 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.178467 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.178471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.178473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.178694 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.179232 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:27.179235 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:27.179358 (+ 123us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.180446 (+ 1088us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.180641 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1741,"thread_start_us":106,"threads_started":1}
I20260810 03:40:27.181684 28702 tablet_copy_service.cc:140] P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 610504ba740c46c88004c6b8611b083e ({username='oryx', principal='oryx/127.21.175.124@KRBTEST.COM'} at 127.0.0.1:58602)
I20260810 03:40:27.181794 28702 tablet_copy_service.cc:161] P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 610504ba740c46c88004c6b8611b083e at {username='oryx', principal='oryx/127.21.175.124@KRBTEST.COM'} at 127.0.0.1:58602: session id = 610504ba740c46c88004c6b8611b083e-00000000000000000000000000000000
I20260810 03:40:27.182876 28702 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 03:40:27.184284 29279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 03:40:27.187147 29279 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: tablet copy: Starting download of 0 data blocks...
I20260810 03:40:27.187372 29279 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: tablet copy: Starting download of 1 WAL segments...
I20260810 03:40:27.189265 29279 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 03:40:27.191425 28702 tablet_copy_service.cc:342] P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Request end of tablet copy session 610504ba740c46c88004c6b8611b083e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.175.124@KRBTEST.COM'} at 127.0.0.1:58602
I20260810 03:40:27.191532 28702 tablet_copy_service.cc:433] P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: ending tablet copy session 610504ba740c46c88004c6b8611b083e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 610504ba740c46c88004c6b8611b083e
I20260810 03:40:27.200309 29279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:27.202251 29488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:27.202548 29279 server_base.cc:1034] running on GCE node
W20260810 03:40:27.202929 29486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:27.206060 29485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:27.206434 29279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:27.206876 29279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:27.208055 29279 hybrid_clock.cc:648] HybridClock initialized: now 1786333227208030 us; error 46 us; skew 500 ppm
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333227, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:27.210072 29279 init.cc:377] Logged in from keytab as oryx/127.21.175.124@KRBTEST.COM (short username oryx)
I20260810 03:40:27.210667 29279 webserver.cc:460] Webserver started at http://127.21.175.124:37233/ using document root <none> and password file <none>
I20260810 03:40:27.210891 29279 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:27.210940 29279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:27.212589 29279 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 03:40:27.213879 29495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:27.214211 29279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 03:40:27.214291 29279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/wal
uuid: "610504ba740c46c88004c6b8611b083e"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "6145d1e2cfc627ee0e5f82c7011abd7d"
server_key_iv: "b1409e568cb30463833bad0ec061fe59"
server_key_version: "encryptionkey@0"
I20260810 03:40:27.214375 29279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:27.224758 29279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:27.227484 29279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:27.233193 29279 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.124:44501
I20260810 03:40:27.233218 29550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.124:44501 every 8 connection(s)
I20260810 03:40:27.233951 29279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/master-2/data/info.pb
I20260810 03:40:27.235881 29551 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:40:27.236989 29551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: Bootstrap starting.
I20260810 03:40:27.249136 29551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:40:27.249509 29551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: Bootstrap complete.
I20260810 03:40:27.250116 29551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true } }
I20260810 03:40:27.250257 29551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 610504ba740c46c88004c6b8611b083e, State: Initialized, Role: LEARNER
I20260810 03:40:27.250355 29551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true } }
I20260810 03:40:27.250805 29551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:27.250741 29558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true } } }
I20260810 03:40:27.251008 29558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [sys.catalog]: This master's current role is: LEARNER
I20260810 03:40:27.254662 29569 catalog_manager.cc:1259] Loaded cluster ID: 523daf2d91334e3999eb5540a2aa6f8c
I20260810 03:40:27.254733 29569 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: loading cluster ID for follower catalog manager: success
W20260810 03:40:27.255065 29469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:27.255398 29469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:27.255479 29469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 03:40:27.256917 29569 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: acquiring CA information for follower catalog manager: success
I20260810 03:40:27.258538 29569 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 03:40:27.260416 29469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:27.260497 29469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:27.260619 29469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:40:27.266479 29469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.65.keytab
--principal=oryx/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:37347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=34813
--tserver_master_addrs=127.21.175.126:36825,127.21.175.125:43497,127.21.175.124:44501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:27.267706 29469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:27.268810 29469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:27.279609 29571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:27.279769 29469 server_base.cc:1034] running on GCE node
W20260810 03:40:27.279611 29572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:27.279692 29574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:27.280439 29469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:27.281180 29469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:27.282372 29469 hybrid_clock.cc:648] HybridClock initialized: now 1786333227282321 us; error 70 us; skew 500 ppm
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333227, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:27.285537 29469 init.cc:377] Logged in from keytab as oryx/127.21.175.65@KRBTEST.COM (short username oryx)
I20260810 03:40:27.286531 29469 webserver.cc:460] Webserver started at http://127.21.175.65:34813/ using document root <none> and password file <none>
I20260810 03:40:27.287145 29469 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:27.287264 29469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:27.291430 29469 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:40:27.293831 29581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:27.295189 29469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:27.295351 29469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "1061eeaa37a74bac8059dae22ab9da04"
format_stamp: "Formatted at 2026-08-10 03:40:25 on dist-test-slave-ss4n"
server_key: "52af0c2de9e50bc2dffd07fe13b93801"
server_key_iv: "872a744622c4848e6e1138aae6a2a585"
server_key_version: "encryptionkey@0"
I20260810 03:40:27.295964 29469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:27.322438 29469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:27.326670 29469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:27.326910 29469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:27.327698 29469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:27.328737 29469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:27.328809 29469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:27.328883 29469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:27.328918 29469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:27.339890 29469 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:37347
I20260810 03:40:27.339975 29704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:37347 every 8 connection(s)
I20260810 03:40:27.341163 29469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:40:27.341770 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 29469
I20260810 03:40:27.341996 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 28890
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333227, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333227, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.125@KRBTEST.COM
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333227, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.124@KRBTEST.COM
I20260810 03:40:27.364503 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:34913
--local_ip_for_outbound_sockets=127.21.175.66
--tserver_master_addrs=127.21.175.126:36825
--webserver_port=36351
--webserver_interface=127.21.175.66
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.66.keytab
--principal=oryx/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.175.126:36825,127.21.175.125:43497
--tserver_master_addrs=127.21.175.126:36825,127.21.175.125:43497,127.21.175.124:44501 with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:40:27.365036 29446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.346201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:50253 (local address 127.21.175.125:43497)
0810 03:40:27.346356 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.346361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.351743 (+ 5382us) server_negotiation.cc:407] Connection header received
0810 03:40:27.351939 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.351943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.352014 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.352137 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.352854 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.353755 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.355150 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.355542 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.355623 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.359247 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.359252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.359254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.359282 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.361430 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.363380 (+ 1950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.363383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.363385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.363466 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.363805 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.363809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.363811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.364038 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.364117 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.364727 (+ 610us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.364858 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:40:27.365376 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.344614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:39833 (local address 127.21.175.126:36825)
0810 03:40:27.344759 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.344764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.346673 (+ 1909us) server_negotiation.cc:407] Connection header received
0810 03:40:27.348130 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.348139 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.348229 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.348362 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.350442 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.351576 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.353977 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.354322 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.354417 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.359805 (+ 5388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.359813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.359820 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.359862 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.362433 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.363057 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.363063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.363068 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.363170 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.363599 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.363611 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.363613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.363876 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.363960 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.365088 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.365218 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:40:27.366104 29710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.344261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:49257 (local address 127.21.175.124:44501)
0810 03:40:27.351004 (+ 6743us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.351008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.351293 (+ 285us) server_negotiation.cc:407] Connection header received
0810 03:40:27.351537 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.351541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.351602 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.351704 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.353167 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.354467 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.355761 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.356206 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.356350 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.360421 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.360427 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.360431 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.360471 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.362473 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.363165 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.363172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.363178 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.363283 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.363638 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.363641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.363645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.363940 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.364051 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.365560 (+ 1509us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.365728 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6645,"thread_start_us":173,"threads_started":1}
I20260810 03:40:27.366392 29712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.345969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:43497 (local address 127.21.175.65:50253)
0810 03:40:27.351646 (+ 5677us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.351663 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.351815 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.352328 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.352332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.352543 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:27.352683 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.352695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.353907 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.353910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.355008 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.355020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.355793 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.355797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.355896 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.356746 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:27.356764 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:27.359015 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:27.361604 (+ 2589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.361613 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.361631 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.363265 (+ 1634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.363605 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.363607 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.363609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.363697 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.364187 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:27.364189 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:27.364408 (+ 219us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.365257 (+ 849us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.365398 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5590,"mutex_wait_us":108,"spinlock_wait_cycles":4864,"thread_start_us":130,"threads_started":1}
I20260810 03:40:27.366739 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.344069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:44501 (local address 127.21.175.65:49257)
0810 03:40:27.351174 (+ 7105us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.351188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.351397 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.351846 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.351853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.352484 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:27.353005 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.353017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.354618 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.354622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.355616 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.355626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.356367 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.356371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.356472 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.357364 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:27.357377 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:27.360246 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:27.362624 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.362631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.362636 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.363019 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.363403 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.363409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.363415 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.363519 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.364106 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:27.364109 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:27.364409 (+ 300us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.364651 (+ 242us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.365102 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7011,"mutex_wait_us":103,"spinlock_wait_cycles":25728,"thread_start_us":361,"threads_started":1}
I20260810 03:40:27.366462 29713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.344069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.65:39833)
0810 03:40:27.346447 (+ 2378us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.346523 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.347849 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.348599 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.348612 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.349242 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:27.350232 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.350256 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.351999 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.352006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.353800 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.353812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.354665 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.354669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.355010 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.355786 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:27.355809 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:27.359583 (+ 3774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:27.362564 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.362568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.362569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.362925 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.363290 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.363296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.363301 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.363484 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.364026 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:27.364033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:27.364408 (+ 375us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.365964 (+ 1556us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.366106 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2102,"mutex_wait_us":52,"spinlock_wait_cycles":17152,"thread_start_us":212,"threads_started":1}
I20260810 03:40:27.368203 29705 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36825
I20260810 03:40:27.368311 29706 heartbeater.cc:347] Connected to a master server at 127.21.175.124:44501
I20260810 03:40:27.368387 29707 heartbeater.cc:347] Connected to a master server at 127.21.175.125:43497
I20260810 03:40:27.368513 29707 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:27.368513 29706 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:27.368513 29705 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:27.369223 29706 heartbeater.cc:511] Master 127.21.175.124:44501 requested a full tablet report, sending...
I20260810 03:40:27.369253 29705 heartbeater.cc:511] Master 127.21.175.126:36825 requested a full tablet report, sending...
I20260810 03:40:27.369253 29707 heartbeater.cc:511] Master 127.21.175.125:43497 requested a full tablet report, sending...
I20260810 03:40:27.370678 28680 ts_manager.cc:194] Re-registered known tserver with Master: 1061eeaa37a74bac8059dae22ab9da04 (127.21.175.65:37347)
I20260810 03:40:27.370965 29315 ts_manager.cc:194] Registered new tserver with Master: 1061eeaa37a74bac8059dae22ab9da04 (127.21.175.65:37347)
I20260810 03:40:27.371137 29513 ts_manager.cc:194] Registered new tserver with Master: 1061eeaa37a74bac8059dae22ab9da04 (127.21.175.65:37347)
I20260810 03:40:27.371446 28680 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:39833
I20260810 03:40:27.379637 29166 heartbeater.cc:503] Master 127.21.175.126:36825 was elected leader, sending a full tablet report...
W20260810 03:40:27.489019 29714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:27.489323 29714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:27.489369 29714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:27.493541 29714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:27.493615 29714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:27.493708 29714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:40:27.499548 29714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.66.keytab
--principal=oryx/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:34913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=36351
--tserver_master_addrs=127.21.175.126:36825,127.21.175.125:43497,127.21.175.124:44501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:27.500808 29714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:27.501979 29714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:27.512723 29719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:27.512709 29722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:27.512713 29720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:27.513142 29714 server_base.cc:1034] running on GCE node
I20260810 03:40:27.513660 29714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:27.514449 29714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:27.515671 29714 hybrid_clock.cc:648] HybridClock initialized: now 1786333227515640 us; error 46 us; skew 500 ppm
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333227, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:27.518913 29714 init.cc:377] Logged in from keytab as oryx/127.21.175.66@KRBTEST.COM (short username oryx)
I20260810 03:40:27.519853 29714 webserver.cc:460] Webserver started at http://127.21.175.66:36351/ using document root <none> and password file <none>
I20260810 03:40:27.520427 29714 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:27.520506 29714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:27.524804 29714 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:27.527308 29729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:27.528832 29714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:27.528995 29714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "4e60af6f969edc9889d5053672224866"
server_key_iv: "b957a64194ebaa9796525b3c5f5cebcd"
server_key_version: "encryptionkey@0"
I20260810 03:40:27.529719 29714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:27.543505 29714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:27.547117 29714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:27.547359 29714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:27.548086 29714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:27.549082 29714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:27.549134 29714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:27.549213 29714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:27.549249 29714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:27.560164 29714 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:34913
I20260810 03:40:27.560187 29850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:34913 every 8 connection(s)
I20260810 03:40:27.561507 29714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:40:27.564054 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 29714
I20260810 03:40:27.564240 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 29031
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333227, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333227, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.124@KRBTEST.COM
I20260810 03:40:27.582518 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:42793
--local_ip_for_outbound_sockets=127.21.175.67
--tserver_master_addrs=127.21.175.126:36825
--webserver_port=42489
--webserver_interface=127.21.175.67
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.67.keytab
--principal=oryx/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.175.126:36825,127.21.175.125:43497
--tserver_master_addrs=127.21.175.126:36825,127.21.175.125:43497,127.21.175.124:44501 with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:40:27.584022 29710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.563555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:40809 (local address 127.21.175.124:44501)
0810 03:40:27.563716 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.563720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.566668 (+ 2948us) server_negotiation.cc:407] Connection header received
0810 03:40:27.567915 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.567921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.568014 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.568147 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.570297 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.571220 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.573529 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.573927 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.574018 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.578726 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.578732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.578736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.578772 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.580796 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.582050 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.582054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.582056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.582171 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.582530 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.582534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.582536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.582760 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.582847 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.583697 (+ 850us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.583843 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 03:40:27.584022 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.566376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:40051 (local address 127.21.175.126:36825)
0810 03:40:27.566512 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.566516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.566748 (+ 232us) server_negotiation.cc:407] Connection header received
0810 03:40:27.568350 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.568357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.568433 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.568539 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.570295 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.571227 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.573528 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.573929 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.574018 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.578184 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.578188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.578194 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.578230 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.580907 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.581775 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.581782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.581785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.581896 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.582401 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.582405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.582406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.582692 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.582791 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.583697 (+ 906us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.583838 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 03:40:27.586045 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.565839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.66:40051)
0810 03:40:27.566669 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.566684 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.567632 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.568583 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.568586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.569045 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:27.570094 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.570117 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.571419 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.571422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.573334 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.573347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.574267 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.574272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.574639 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.575581 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:27.575619 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:27.577971 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:27.581010 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.581014 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.581017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.581647 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.582025 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.582030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.582032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.582277 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.583237 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:27.583240 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:27.583357 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.584650 (+ 1293us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.584830 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":396,"spinlock_wait_cycles":20736,"thread_start_us":271,"threads_started":1}
I20260810 03:40:27.586045 29857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.564046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:44501 (local address 127.21.175.66:40809)
0810 03:40:27.566480 (+ 2434us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.566549 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.567632 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.568319 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.568327 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.569045 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:27.570094 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.570117 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.571469 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.571476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.573334 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.573347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.574267 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.574272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.574643 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.575669 (+ 1026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:27.575685 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:27.578543 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:27.580961 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.580972 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.580997 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.581910 (+ 913us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.582302 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.582309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.582314 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.582410 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.582956 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:27.582966 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:27.583357 (+ 391us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.584286 (+ 929us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.584769 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2134,"mutex_wait_us":450,"spinlock_wait_cycles":15232,"thread_start_us":1177,"threads_started":1}
I20260810 03:40:27.587795 29851 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36825
I20260810 03:40:27.588101 29851 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:27.588653 29853 heartbeater.cc:347] Connected to a master server at 127.21.175.124:44501
I20260810 03:40:27.588702 29851 heartbeater.cc:511] Master 127.21.175.126:36825 requested a full tablet report, sending...
I20260810 03:40:27.588761 29853 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:27.588966 29853 heartbeater.cc:511] Master 127.21.175.124:44501 requested a full tablet report, sending...
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333227, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.125@KRBTEST.COM
I20260810 03:40:27.590075 28680 ts_manager.cc:194] Re-registered known tserver with Master: 0dda5f21ab0d4dd4bdef3d73df6bd04c (127.21.175.66:34913)
I20260810 03:40:27.590167 29513 ts_manager.cc:194] Registered new tserver with Master: 0dda5f21ab0d4dd4bdef3d73df6bd04c (127.21.175.66:34913)
I20260810 03:40:27.591043 28680 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:40051
I20260810 03:40:27.594034 29446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.581685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:47119 (local address 127.21.175.125:43497)
0810 03:40:27.581854 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.581860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.582302 (+ 442us) server_negotiation.cc:407] Connection header received
0810 03:40:27.582495 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.582499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.582554 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.582624 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.583351 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.584181 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.585559 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.585875 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.585937 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.589923 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.589927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.589930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.589961 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.591951 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.592591 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.592594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.592597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.592701 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.593067 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.593071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.593072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.593318 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.593442 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.593790 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.593903 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:40:27.594192 29858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.566566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:43497 (local address 127.21.175.66:47119)
0810 03:40:27.582204 (+ 15638us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.582220 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.582375 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.582805 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.582810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.583023 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:27.583205 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.583216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.584348 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.584352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.585381 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.585396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.586086 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.586089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.586206 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.587916 (+ 1710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:27.587936 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:27.589748 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:27.592139 (+ 2391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.592144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.592149 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.592442 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.592822 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.592828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.592832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.592941 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.593440 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:27.593445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:27.593590 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.593711 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.593917 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15548,"mutex_wait_us":6,"thread_start_us":160,"threads_started":1}
I20260810 03:40:27.594942 29854 heartbeater.cc:347] Connected to a master server at 127.21.175.125:43497
I20260810 03:40:27.595085 29854 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:27.595352 29854 heartbeater.cc:511] Master 127.21.175.125:43497 requested a full tablet report, sending...
I20260810 03:40:27.596458 29315 ts_manager.cc:194] Registered new tserver with Master: 0dda5f21ab0d4dd4bdef3d73df6bd04c (127.21.175.66:34913)
I20260810 03:40:27.705520 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.700805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:44501 (local address 127.0.0.1:42802)
0810 03:40:27.701120 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.701135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.701245 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.701689 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.701693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.701718 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:27.701844 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.701850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.702912 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.702915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.704147 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.704159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.705026 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.705030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.705129 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.705159 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.705247 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.705336 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
I20260810 03:40:27.705513 29710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.700936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42802 (local address 127.21.175.124:44501)
0810 03:40:27.701098 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.701103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.701222 (+ 119us) server_negotiation.cc:407] Connection header received
0810 03:40:27.701351 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.701354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.701513 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.701668 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:27.702016 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.702790 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.704327 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.704845 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.704960 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.705147 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.705260 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.705321 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 03:40:27.706204 29523 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 03:40:27.706622 29558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69c0ef9dc48b4bb5a29c2ca3b4eb44ca. Latest consensus state: current_term: 1 leader_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true } } }
I20260810 03:40:27.706795 29558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [sys.catalog]: This master's current role is: LEARNER
W20260810 03:40:27.708336 29859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:27.708755 29859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:27.708861 29859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:27.713167 29859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:27.713235 29859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:27.713341 29859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:40:27.719306 29859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.67.keytab
--principal=oryx/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:42793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=42489
--tserver_master_addrs=127.21.175.126:36825,127.21.175.125:43497,127.21.175.124:44501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:27.720544 29859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:27.721757 29859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:27.732802 29866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:27.732810 29867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:27.732810 29869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:27.733109 29859 server_base.cc:1034] running on GCE node
I20260810 03:40:27.733624 29859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:27.734393 29859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:27.735576 29859 hybrid_clock.cc:648] HybridClock initialized: now 1786333227735554 us; error 38 us; skew 500 ppm
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333227, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:27.739216 29859 init.cc:377] Logged in from keytab as oryx/127.21.175.67@KRBTEST.COM (short username oryx)
I20260810 03:40:27.740186 29859 webserver.cc:460] Webserver started at http://127.21.175.67:42489/ using document root <none> and password file <none>
I20260810 03:40:27.740799 29859 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:27.740906 29859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:27.745219 29859 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:27.747563 29876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:27.749040 29859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:27.749188 29859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "9c144606813640a981f6128257243438"
format_stamp: "Formatted at 2026-08-10 03:40:26 on dist-test-slave-ss4n"
server_key: "47d08e4df635be0e276e3927188173b5"
server_key_iv: "fb4bdf8f8a9bfa7f6781047cce20dfed"
server_key_version: "encryptionkey@0"
I20260810 03:40:27.749840 29859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:27.766450 29859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:27.770100 29859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:27.770328 29859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:27.770984 29859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:27.771907 29859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:27.771978 29859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:27.772060 29859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:27.772111 29859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:27.781661 29859 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:42793
I20260810 03:40:27.781741 29995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:42793 every 8 connection(s)
I20260810 03:40:27.782796 29859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:40:27.792217 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 29859
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333227, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.125@KRBTEST.COM
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333227, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM
Aug 10 03:40:27 dist-test-slave-ss4n krb5kdc[28633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333227, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.124@KRBTEST.COM
I20260810 03:40:27.802461 30001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.785211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.67:55571)
0810 03:40:27.785791 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.785848 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.787044 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.787699 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.787709 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.788337 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:27.789023 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.789038 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.790028 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.790032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.791464 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.791477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.792010 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.792014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.792177 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.792961 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:27.792987 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:27.796205 (+ 3218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:27.798573 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.798583 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.798605 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.799036 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.799469 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.799475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.799477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.799658 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.800153 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:27.800160 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:27.800601 (+ 441us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.800824 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.801214 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":168,"spinlock_wait_cycles":14592,"thread_start_us":136,"threads_started":1}
I20260810 03:40:27.803207 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.784794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:55571 (local address 127.21.175.126:36825)
0810 03:40:27.784960 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.784967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.785966 (+ 999us) server_negotiation.cc:407] Connection header received
0810 03:40:27.787302 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.787306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.787375 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.787489 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.789189 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.789856 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.791602 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.791915 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.792005 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.796329 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.796338 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.796339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.796383 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.798436 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.799263 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.799267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.799269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.799361 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.799767 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.799770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.799772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.799982 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.800098 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.802914 (+ 2816us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.803059 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:40:27.803761 29446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.786530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:38071 (local address 127.21.175.125:43497)
0810 03:40:27.786657 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.786662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.786813 (+ 151us) server_negotiation.cc:407] Connection header received
0810 03:40:27.787655 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.787658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.787709 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.787801 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.789196 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.790043 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.792105 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.792351 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.792451 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.795526 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.795534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.795540 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.795579 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.801268 (+ 5689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.802242 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.802248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.802254 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.802374 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.802751 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.802755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.802756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.803008 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.803081 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.803464 (+ 383us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.803593 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 03:40:27.803761 29710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.784739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:47091 (local address 127.21.175.124:44501)
0810 03:40:27.784874 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.784879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.785965 (+ 1086us) server_negotiation.cc:407] Connection header received
0810 03:40:27.787445 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.787448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.787510 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.787604 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.789196 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.790083 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.791338 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.791622 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.791725 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.798648 (+ 6923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.798654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.798660 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.798693 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.800589 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.802291 (+ 1702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.802297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.802298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.802387 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.802756 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.802760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.802763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.802984 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.803081 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.803464 (+ 383us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.803593 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 03:40:27.804355 30002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.785211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:44501 (local address 127.21.175.67:47091)
0810 03:40:27.785791 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.785847 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.787039 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.787699 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.787709 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.788342 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:27.789023 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.789040 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.790228 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.790230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.791195 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.791204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.791773 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.791776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.792555 (+ 779us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.793122 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:27.793142 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:27.797892 (+ 4750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:27.801398 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.801404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.801410 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.801733 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.802511 (+ 778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.802513 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.802516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.802640 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.803150 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:27.803153 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:27.803255 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.803974 (+ 719us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.804136 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":212,"spinlock_wait_cycles":134016,"thread_start_us":185,"threads_started":1}
I20260810 03:40:27.805543 29997 heartbeater.cc:347] Connected to a master server at 127.21.175.124:44501
I20260810 03:40:27.805619 29996 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36825
I20260810 03:40:27.805820 29997 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:27.805820 29996 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:27.806321 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.786344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:43497 (local address 127.21.175.67:38071)
0810 03:40:27.786715 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.786728 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.787040 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.787833 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.787837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.788342 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:27.789023 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.789040 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.790161 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.790164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.791472 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.791479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.792630 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.792635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.792742 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.793289 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:27.793304 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:27.795292 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:27.801846 (+ 6554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.801849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.801851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.802092 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.802504 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:27.802508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:27.802510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:27.802624 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:27.803181 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:27.803184 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:27.803256 (+ 72us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.805886 (+ 2630us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.806058 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":278,"spinlock_wait_cycles":9216,"thread_start_us":97,"threads_started":1}
I20260810 03:40:27.806489 29997 heartbeater.cc:511] Master 127.21.175.124:44501 requested a full tablet report, sending...
I20260810 03:40:27.806674 29996 heartbeater.cc:511] Master 127.21.175.126:36825 requested a full tablet report, sending...
I20260810 03:40:27.807006 29998 heartbeater.cc:347] Connected to a master server at 127.21.175.125:43497
I20260810 03:40:27.807086 29998 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:27.807271 29998 heartbeater.cc:511] Master 127.21.175.125:43497 requested a full tablet report, sending...
I20260810 03:40:27.807662 28680 ts_manager.cc:194] Re-registered known tserver with Master: 9c144606813640a981f6128257243438 (127.21.175.67:42793)
I20260810 03:40:27.807761 29513 ts_manager.cc:194] Registered new tserver with Master: 9c144606813640a981f6128257243438 (127.21.175.67:42793)
I20260810 03:40:27.808292 29315 ts_manager.cc:194] Registered new tserver with Master: 9c144606813640a981f6128257243438 (127.21.175.67:42793)
I20260810 03:40:27.808495 28680 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:55571
I20260810 03:40:27.812415 30011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.795514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41776 (local address 127.21.175.125:43497)
0810 03:40:27.801254 (+ 5740us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.801259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.801274 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:40:27.801312 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.801318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.801378 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.801468 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.803111 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.803898 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.805052 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.805442 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.805618 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.807970 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.807974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.807976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.808002 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.810406 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.810953 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.810956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.810958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.811065 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.811366 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.811370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.811372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.811599 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.811733 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.812058 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.812192 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5639,"thread_start_us":107,"threads_started":1}
I20260810 03:40:27.812527 30009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.794953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.21.175.126:36825)
0810 03:40:27.795214 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.795228 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.796400 (+ 1172us) server_negotiation.cc:407] Connection header received
0810 03:40:27.796552 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.796558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.796618 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.796711 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.797868 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.798475 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.800107 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.800441 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.800527 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.805129 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.805132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.805135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.805163 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.810215 (+ 5052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.810918 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.810923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.810925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.811031 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.811446 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.811450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.811451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.811681 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.811795 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.812186 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.812334 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
I20260810 03:40:27.815068 30013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.795911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42806 (local address 127.21.175.124:44501)
0810 03:40:27.796164 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.796168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.797320 (+ 1152us) server_negotiation.cc:407] Connection header received
0810 03:40:27.797379 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.797384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.797446 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.797559 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:27.798695 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.799642 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.801066 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.801371 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.801469 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.810763 (+ 9294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:27.810770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:27.810775 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:27.810800 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:27.812822 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.813536 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.813540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.813542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.813645 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:27.814001 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:27.814004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:27.814006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:27.814295 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:27.814432 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.814682 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.814830 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":106,"threads_started":1}
I20260810 03:40:27.816073 28680 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:40:27.819533 28680 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:40:27.822661 30016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: NON_VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: true } } }
I20260810 03:40:27.822901 30016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:27.826759 30016 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: attempting to promote NON_VOTER 610504ba740c46c88004c6b8611b083e to VOTER
I20260810 03:40:27.827075 30016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } }
I20260810 03:40:27.828657 29327 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 FOLLOWER]: Refusing update from remote peer 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 03:40:27.828682 29523 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 1 LEARNER]: Refusing update from remote peer 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 03:40:27.829118 30017 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 03:40:27.829411 30017 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 03:40:27.831226 30017 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 610504ba740c46c88004c6b8611b083e (127.21.175.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } } }
I20260810 03:40:27.831573 29327 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 610504ba740c46c88004c6b8611b083e (127.21.175.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } } }
I20260810 03:40:27.832356 29523 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 610504ba740c46c88004c6b8611b083e (127.21.175.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } } }
I20260810 03:40:27.833045 30017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } } }
I20260810 03:40:27.833155 30017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:27.833282 30017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } } }
I20260810 03:40:27.833353 30017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:27.834930 29558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } } }
I20260810 03:40:27.835060 29558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:40:27.835260 30026 sys_catalog.cc:455] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } } }
I20260810 03:40:27.835393 30026 sys_catalog.cc:458] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:40:27.833446 30017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } } }
I20260810 03:40:27.835436 30017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:27.847754 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.841978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:42793 (local address 127.0.0.1:41048)
0810 03:40:27.842170 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.842188 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.842278 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.843299 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.843303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.843323 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:27.843443 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.843450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.844737 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.844741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.846032 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.846043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.847318 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.847322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.847436 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.847467 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.847522 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.847604 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260810 03:40:27.848531 30027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.842225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41048 (local address 127.21.175.67:42793)
0810 03:40:27.842708 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.842725 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.842753 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:40:27.842840 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.842847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.843130 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.843313 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:27.843577 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.844615 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.846174 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.847169 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.848000 (+ 831us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.848177 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.848240 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.848353 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":176,"threads_started":1}
I20260810 03:40:27.848874 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.842344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:37347 (local address 127.0.0.1:35922)
0810 03:40:27.843289 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.843305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.843416 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.844028 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.844032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.844053 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:27.844158 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.844165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.845555 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.845558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.846870 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.846883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.848346 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.848350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.848449 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.848482 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.848583 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.848673 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260810 03:40:27.849643 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.842806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35922 (local address 127.21.175.65:37347)
0810 03:40:27.843177 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.843203 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.843525 (+ 322us) server_negotiation.cc:407] Connection header received
0810 03:40:27.843620 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.843626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.843842 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.844036 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:27.844271 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.845385 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.847060 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.848213 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.848331 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.848475 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.848804 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.848926 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":138,"threads_started":1}
I20260810 03:40:27.850422 30028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.843045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:34913 (local address 127.0.0.1:34182)
0810 03:40:27.844817 (+ 1772us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.844831 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.844937 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.845435 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.845440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.845465 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:27.845632 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.845641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.846786 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.846791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.847972 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.847986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.850022 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.850025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.850094 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.850116 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.850210 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.850289 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1708,"mutex_wait_us":26,"thread_start_us":99,"threads_started":1}
I20260810 03:40:27.850703 30032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.843921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.21.175.66:34913)
0810 03:40:27.844654 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.844666 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.844939 (+ 273us) server_negotiation.cc:407] Connection header received
0810 03:40:27.845053 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.845059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.845260 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.845427 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:27.845761 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.846663 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.848086 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.849667 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.849812 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.849962 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.850258 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.850360 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":108,"threads_started":1}
I20260810 03:40:27.852196 29927 tablet_service.cc:1467] Processing CreateTablet for tablet aa1841b398aa42c482324a3128dce34c (DEFAULT_TABLE table=test-table [id=12c48a87002d4bb38c7bef39834c1e44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:27.852545 29633 tablet_service.cc:1467] Processing CreateTablet for tablet aa1841b398aa42c482324a3128dce34c (DEFAULT_TABLE table=test-table [id=12c48a87002d4bb38c7bef39834c1e44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:27.853574 29633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1841b398aa42c482324a3128dce34c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:27.854400 29927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1841b398aa42c482324a3128dce34c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:27.856395 29780 tablet_service.cc:1467] Processing CreateTablet for tablet aa1841b398aa42c482324a3128dce34c (DEFAULT_TABLE table=test-table [id=12c48a87002d4bb38c7bef39834c1e44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:27.857539 29780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1841b398aa42c482324a3128dce34c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:27.863224 30034 tablet_bootstrap.cc:492] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438: Bootstrap starting.
I20260810 03:40:27.865638 30035 tablet_bootstrap.cc:492] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c: Bootstrap starting.
I20260810 03:40:27.866421 30034 tablet_bootstrap.cc:654] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:27.867359 30034 log.cc:824] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:27.867905 30035 tablet_bootstrap.cc:654] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:27.868193 30033 tablet_bootstrap.cc:492] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04: Bootstrap starting.
I20260810 03:40:27.868947 30035 log.cc:824] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:27.869993 30034 tablet_bootstrap.cc:492] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438: No bootstrap required, opened a new log
I20260810 03:40:27.870222 30034 ts_tablet_manager.cc:1397] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 03:40:27.871322 30033 tablet_bootstrap.cc:654] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:27.872291 30033 log.cc:824] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:27.874339 30033 tablet_bootstrap.cc:492] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04: No bootstrap required, opened a new log
I20260810 03:40:27.874475 30035 tablet_bootstrap.cc:492] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c: No bootstrap required, opened a new log
I20260810 03:40:27.874614 30033 ts_tablet_manager.cc:1397] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 03:40:27.874698 30035 ts_tablet_manager.cc:1397] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20260810 03:40:27.875738 30034 raft_consensus.cc:348] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } }
I20260810 03:40:27.876062 30034 raft_consensus.cc:374] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:27.876118 30034 raft_consensus.cc:729] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c144606813640a981f6128257243438, State: Initialized, Role: FOLLOWER
I20260810 03:40:27.876703 30034 consensus_queue.cc:260] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } }
I20260810 03:40:27.877584 29996 heartbeater.cc:503] Master 127.21.175.126:36825 was elected leader, sending a full tablet report...
I20260810 03:40:27.877986 30034 ts_tablet_manager.cc:1428] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260810 03:40:27.878423 30033 raft_consensus.cc:348] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } }
I20260810 03:40:27.878701 30033 raft_consensus.cc:374] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:27.878755 30033 raft_consensus.cc:729] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1061eeaa37a74bac8059dae22ab9da04, State: Initialized, Role: FOLLOWER
I20260810 03:40:27.879252 30033 consensus_queue.cc:260] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } }
I20260810 03:40:27.880429 30033 ts_tablet_manager.cc:1428] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 03:40:27.880832 29705 heartbeater.cc:503] Master 127.21.175.126:36825 was elected leader, sending a full tablet report...
I20260810 03:40:27.883066 30035 raft_consensus.cc:348] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } }
I20260810 03:40:27.883400 30035 raft_consensus.cc:374] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:27.883489 30035 raft_consensus.cc:729] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dda5f21ab0d4dd4bdef3d73df6bd04c, State: Initialized, Role: FOLLOWER
I20260810 03:40:27.884128 30035 consensus_queue.cc:260] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } }
I20260810 03:40:27.885032 29851 heartbeater.cc:503] Master 127.21.175.126:36825 was elected leader, sending a full tablet report...
I20260810 03:40:27.885388 30035 ts_tablet_manager.cc:1428] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c: Time spent starting tablet: real 0.011s user 0.006s sys 0.002s
I20260810 03:40:27.959283 30042 raft_consensus.cc:482] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:27.959570 30042 raft_consensus.cc:504] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } }
I20260810 03:40:27.960903 30042 leader_election.cc:288] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c144606813640a981f6128257243438 (127.21.175.67:42793), 1061eeaa37a74bac8059dae22ab9da04 (127.21.175.65:37347)
I20260810 03:40:27.964711 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.961545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:51149 (local address 127.21.175.65:37347)
0810 03:40:27.961690 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.961697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.961714 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:40:27.961778 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.961780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.961816 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.961904 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:27.962210 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.962713 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.963730 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.964132 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.964302 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.964375 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.964516 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.964584 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:40:27.964733 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.961201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:37347 (local address 127.21.175.66:51149)
0810 03:40:27.961505 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.961530 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.961642 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.961920 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.961923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.961939 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:27.962084 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.962090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.962834 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.962840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.963597 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.963613 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.964269 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.964273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.964376 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.964405 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.964500 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.964577 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":158}
I20260810 03:40:27.965231 30027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.961525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:59841 (local address 127.21.175.67:42793)
0810 03:40:27.961756 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:27.961760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:27.961781 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:40:27.961839 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:27.961841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:27.961885 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:27.961995 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:27.962355 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.962925 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.964074 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.964549 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.964706 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.964767 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:27.965024 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 03:40:27.965114 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260810 03:40:27.965255 29858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:27.961202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:42793 (local address 127.21.175.66:59841)
0810 03:40:27.961393 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:27.961416 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:27.961578 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:27.962022 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:27.962026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:27.962046 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:27.962209 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.962216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.963060 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.963067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.963906 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:27.963916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:27.964695 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:27.964700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:27.964831 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:27.964871 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 03:40:27.964987 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 03:40:27.965063 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 03:40:27.965677 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1841b398aa42c482324a3128dce34c" candidate_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1061eeaa37a74bac8059dae22ab9da04" is_pre_election: true
I20260810 03:40:27.966033 29947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1841b398aa42c482324a3128dce34c" candidate_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c144606813640a981f6128257243438" is_pre_election: true
I20260810 03:40:27.966091 29653 raft_consensus.cc:2454] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dda5f21ab0d4dd4bdef3d73df6bd04c in term 0.
I20260810 03:40:27.966355 29947 raft_consensus.cc:2454] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dda5f21ab0d4dd4bdef3d73df6bd04c in term 0.
I20260810 03:40:27.966704 29730 leader_election.cc:302] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dda5f21ab0d4dd4bdef3d73df6bd04c, 1061eeaa37a74bac8059dae22ab9da04; no voters:
I20260810 03:40:27.967164 30042 raft_consensus.cc:2781] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:40:27.967234 30042 raft_consensus.cc:482] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:40:27.967263 30042 raft_consensus.cc:3037] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:27.968331 30042 raft_consensus.cc:504] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } }
I20260810 03:40:27.968909 30042 leader_election.cc:288] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [CANDIDATE]: Term 1 election: Requested vote from peers 9c144606813640a981f6128257243438 (127.21.175.67:42793), 1061eeaa37a74bac8059dae22ab9da04 (127.21.175.65:37347)
I20260810 03:40:27.969260 29947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1841b398aa42c482324a3128dce34c" candidate_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c144606813640a981f6128257243438"
I20260810 03:40:27.969379 29947 raft_consensus.cc:3037] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:27.969520 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1841b398aa42c482324a3128dce34c" candidate_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1061eeaa37a74bac8059dae22ab9da04"
I20260810 03:40:27.969676 29653 raft_consensus.cc:3037] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:27.970817 29947 raft_consensus.cc:2454] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dda5f21ab0d4dd4bdef3d73df6bd04c in term 1.
I20260810 03:40:27.970996 29653 raft_consensus.cc:2454] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dda5f21ab0d4dd4bdef3d73df6bd04c in term 1.
I20260810 03:40:27.971151 29732 leader_election.cc:302] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dda5f21ab0d4dd4bdef3d73df6bd04c, 9c144606813640a981f6128257243438; no voters:
I20260810 03:40:27.971330 30042 raft_consensus.cc:2781] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:27.971565 30042 raft_consensus.cc:686] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 1 LEADER]: Becoming Leader. State: Replica: 0dda5f21ab0d4dd4bdef3d73df6bd04c, State: Running, Role: LEADER
I20260810 03:40:27.971936 30042 consensus_queue.cc:237] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } }
I20260810 03:40:27.975595 28676 catalog_manager.cc:5495] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dda5f21ab0d4dd4bdef3d73df6bd04c (127.21.175.66). New cstate: current_term: 1 leader_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } health_report { overall_health: HEALTHY } } }
I20260810 03:40:28.007258 30032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:28.002136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34198 (local address 127.21.175.66:34913)
0810 03:40:28.002300 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:28.002304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:28.002940 (+ 636us) server_negotiation.cc:407] Connection header received
0810 03:40:28.003113 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:28.003115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:28.003173 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:28.003268 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:40:28.003735 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:28.004334 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:28.005684 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:28.005962 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:28.006063 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:28.006381 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:40:28.006858 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:40:28.006956 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:28.007018 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 03:40:28.007076 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 03:40:28.013394 29653 raft_consensus.cc:1261] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04 [term 1 FOLLOWER]: Refusing update from remote peer 0dda5f21ab0d4dd4bdef3d73df6bd04c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:28.013393 29947 raft_consensus.cc:1261] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438 [term 1 FOLLOWER]: Refusing update from remote peer 0dda5f21ab0d4dd4bdef3d73df6bd04c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:28.014245 30043 consensus_queue.cc:1035] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:28.014448 30042 consensus_queue.cc:1035] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:28.034462 28676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:28.034720 28676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:28.041554 28676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69c0ef9dc48b4bb5a29c2ca3b4eb44ca: Sending DeleteTablet for 3 replicas of tablet aa1841b398aa42c482324a3128dce34c
I20260810 03:40:28.042537 29927 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1841b398aa42c482324a3128dce34c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:28 UTC) from {username='oryx', principal='oryx/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:41048
I20260810 03:40:28.042680 29633 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1841b398aa42c482324a3128dce34c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:28 UTC) from {username='oryx', principal='oryx/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:35922
I20260810 03:40:28.042848 29780 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1841b398aa42c482324a3128dce34c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:28 UTC) from {username='oryx', principal='oryx/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:34182
I20260810 03:40:28.043331 30058 tablet_replica.cc:330] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c: stopping tablet replica
I20260810 03:40:28.043747 30058 raft_consensus.cc:2229] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:40:28.044080 30056 tablet_replica.cc:330] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438: stopping tablet replica
I20260810 03:40:28.044193 30058 raft_consensus.cc:2258] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:28.044561 30056 raft_consensus.cc:2229] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:28.044987 30056 raft_consensus.cc:2258] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:28.045850 30057 tablet_replica.cc:330] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04: stopping tablet replica
I20260810 03:40:28.046615 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 28648
I20260810 03:40:28.048153 30058 ts_tablet_manager.cc:1905] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:28.050086 30056 ts_tablet_manager.cc:1905] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:28.051354 30058 ts_tablet_manager.cc:1918] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:40:28.051522 30058 log.cc:1196] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/aa1841b398aa42c482324a3128dce34c
I20260810 03:40:28.051946 30058 ts_tablet_manager.cc:1939] T aa1841b398aa42c482324a3128dce34c P 0dda5f21ab0d4dd4bdef3d73df6bd04c: Deleting consensus metadata
I20260810 03:40:28.053520 30056 ts_tablet_manager.cc:1918] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:40:28.053730 30056 log.cc:1196] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/aa1841b398aa42c482324a3128dce34c
I20260810 03:40:28.054159 30056 ts_tablet_manager.cc:1939] T aa1841b398aa42c482324a3128dce34c P 9c144606813640a981f6128257243438: Deleting consensus metadata
I20260810 03:40:28.054912 30057 raft_consensus.cc:2229] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:28.056460 30057 raft_consensus.cc:2258] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:28.058509 30057 ts_tablet_manager.cc:1905] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 03:40:28.058660 29733 connection.cc:535] server connection from 127.0.0.1:34182 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 03:40:28.059420 29880 connection.cc:535] server connection from 127.0.0.1:41048 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 03:40:28.062237 30057 ts_tablet_manager.cc:1918] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:40:28.062453 30057 log.cc:1196] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-0/wal/wals/aa1841b398aa42c482324a3128dce34c
I20260810 03:40:28.062918 30057 ts_tablet_manager.cc:1939] T aa1841b398aa42c482324a3128dce34c P 1061eeaa37a74bac8059dae22ab9da04: Deleting consensus metadata
I20260810 03:40:28.064095 30011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:28.045073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.21.175.125:43497)
0810 03:40:28.045208 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:28.045215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:28.045229 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:40:28.045270 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:28.045273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:28.045321 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:28.045413 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:28.054824 (+ 9411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:28.055619 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:28.056838 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:28.057192 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:28.057287 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:28.059928 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:28.059933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:28.059936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:28.059972 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:28.061939 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:28.062559 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:28.062563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:28.062565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:28.062668 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:28.063032 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:28.063036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:28.063037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:28.063294 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:28.063404 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:28.063779 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 03:40:28.063937 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 03:40:28.064533 29586 connection.cc:441] server connection from 127.0.0.1:35922 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35922 (request call id 1) could send its response
I20260810 03:40:28.075968 29446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:28.064130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41798 (local address 127.21.175.125:43497)
0810 03:40:28.064257 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:28.064261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:28.065537 (+ 1276us) server_negotiation.cc:407] Connection header received
0810 03:40:28.065750 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:28.065756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:28.065799 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:28.065867 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:28.066829 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:28.067648 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:28.069021 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:28.069373 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:28.069449 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:28.071933 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:28.071937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:28.071939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:28.071968 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:28.073850 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:28.074395 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:28.074400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:28.074404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:28.074498 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:28.074839 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:28.074843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:28.074845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:28.075128 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:28.075243 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:28.075633 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 03:40:28.075772 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:40:28.077026 29710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:28.064860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.21.175.124:44501)
0810 03:40:28.064983 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:28.064988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:28.066145 (+ 1157us) server_negotiation.cc:407] Connection header received
0810 03:40:28.066513 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:28.066517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:28.066570 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:28.066626 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:28.067665 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:28.068529 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:28.070034 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:28.070413 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:28.070495 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:28.072365 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:28.072370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:28.072373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:28.072401 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:28.074817 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:28.075468 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:28.075472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:28.075474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:28.075588 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:28.075940 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:28.075944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:28.075946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:28.076208 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:28.076321 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:28.076711 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 03:40:28.076843 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:40:28.078119 30013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:28.045263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42820 (local address 127.21.175.124:44501)
0810 03:40:28.045454 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:28.045459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:28.065526 (+ 20067us) server_negotiation.cc:407] Connection header received
0810 03:40:28.065623 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:28.065628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:28.065691 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:28.065768 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:28.066833 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:28.067650 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:28.069021 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:28.069373 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:28.069449 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:28.071767 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:28.071771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:28.071774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:28.071802 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:28.074881 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:28.075472 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:28.075476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:28.075477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:28.075627 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:28.075960 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:28.075963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:28.075965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:28.077047 (+ 1082us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:28.077407 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:28.077796 (+ 389us) server_negotiation.cc:299] Negotiation successful
0810 03:40:28.077918 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"spinlock_wait_cycles":8448}
I20260810 03:40:29.028415 30067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:29.027833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.67:56147)
0810 03:40:29.028145 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:29.028262 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":100,"threads_started":1}
W20260810 03:40:29.028738 29996 heartbeater.cc:644] Failed to heartbeat to 127.21.175.126:36825 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
I20260810 03:40:29.032982 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:29.032460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.65:38389)
0810 03:40:29.032723 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:29.032854 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":95,"threads_started":1}
W20260810 03:40:29.033349 29705 heartbeater.cc:644] Failed to heartbeat to 127.21.175.126:36825 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
I20260810 03:40:29.033871 30069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:29.033336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.66:53901)
0810 03:40:29.033652 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:29.033731 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":101,"threads_started":1}
W20260810 03:40:29.034176 29851 heartbeater.cc:644] Failed to heartbeat to 127.21.175.126:36825 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
I20260810 03:40:29.542105 30070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69c0ef9dc48b4bb5a29c2ca3b4eb44ca)
I20260810 03:40:29.542290 30071 raft_consensus.cc:482] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69c0ef9dc48b4bb5a29c2ca3b4eb44ca)
I20260810 03:40:29.542266 30070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } }
I20260810 03:40:29.542565 30071 raft_consensus.cc:504] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } }
I20260810 03:40:29.542938 30070 leader_election.cc:288] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 69c0ef9dc48b4bb5a29c2ca3b4eb44ca (127.21.175.126:36825), e889ae8ffce6406d80d7ed13193faeb1 (127.21.175.125:43497)
I20260810 03:40:29.543759 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:29.543159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:59102)
0810 03:40:29.543545 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:29.543651 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":66,"threads_started":1}
I20260810 03:40:29.543793 30071 leader_election.cc:288] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 69c0ef9dc48b4bb5a29c2ca3b4eb44ca (127.21.175.126:36825), 610504ba740c46c88004c6b8611b083e (127.21.175.124:44501)
W20260810 03:40:29.544088 29499 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
I20260810 03:40:29.545675 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:29.545363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:59136)
0810 03:40:29.545530 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:29.545584 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260810 03:40:29.545869 29499 leader_election.cc:334] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 69c0ef9dc48b4bb5a29c2ca3b4eb44ca (127.21.175.126:36825): Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
I20260810 03:40:29.546129 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:29.543948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:59128)
0810 03:40:29.545873 (+ 1925us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:29.546013 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1863,"mutex_wait_us":351,"thread_start_us":68,"threads_started":1}
W20260810 03:40:29.546326 29302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
I20260810 03:40:29.547251 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:29.547011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:59146)
0810 03:40:29.547131 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:29.547185 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 03:40:29.547415 29302 leader_election.cc:334] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 69c0ef9dc48b4bb5a29c2ca3b4eb44ca (127.21.175.126:36825): Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
I20260810 03:40:29.548666 30074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:29.543519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41812 (local address 127.21.175.125:43497)
0810 03:40:29.544019 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:29.544032 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:29.544382 (+ 350us) server_negotiation.cc:407] Connection header received
0810 03:40:29.544439 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:29.544442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:29.544511 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:29.544595 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:29.545426 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:29.546285 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:29.547707 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:29.548133 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:29.548255 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:29.548308 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:29.548476 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 03:40:29.548543 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"spinlock_wait_cycles":14976,"thread_start_us":77,"threads_started":1}
I20260810 03:40:29.548751 30076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:29.544188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.21.175.124:44501)
0810 03:40:29.544688 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:29.544693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:29.544707 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:40:29.544740 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:29.544743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:29.544791 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:29.544868 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:29.545209 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:29.546204 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:29.547758 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:29.548137 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:29.548261 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:29.548312 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:29.548454 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 03:40:29.548586 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":438,"spinlock_wait_cycles":5376,"thread_start_us":64,"threads_started":1}
I20260810 03:40:29.548669 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:29.543948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:44501 (local address 127.0.0.1:42840)
0810 03:40:29.544512 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:29.544527 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:29.544628 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:29.544954 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:29.544957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:29.544971 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:29.545073 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:29.545081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:29.546334 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:29.546341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:29.547623 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:29.547641 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:29.548252 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:29.548255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:29.548336 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:29.548353 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 03:40:29.548446 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 03:40:29.548520 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"spinlock_wait_cycles":1920,"thread_start_us":72,"threads_started":1}
I20260810 03:40:29.548751 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:29.543175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:43497 (local address 127.0.0.1:41812)
0810 03:40:29.544150 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:29.544166 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:29.544284 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:29.544741 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:29.544744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:29.544767 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:29.545306 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:29.545315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:29.546384 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:29.546387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:29.547576 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:29.547595 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:29.548261 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:29.548264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:29.548357 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:29.548375 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 03:40:29.548450 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 03:40:29.548599 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":45,"spinlock_wait_cycles":896,"thread_start_us":59,"threads_started":1}
I20260810 03:40:29.549448 29327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "610504ba740c46c88004c6b8611b083e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e889ae8ffce6406d80d7ed13193faeb1" is_pre_election: true
I20260810 03:40:29.549463 29523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e889ae8ffce6406d80d7ed13193faeb1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "610504ba740c46c88004c6b8611b083e" is_pre_election: true
I20260810 03:40:29.549670 29327 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 610504ba740c46c88004c6b8611b083e in term 1.
I20260810 03:40:29.549742 29523 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e889ae8ffce6406d80d7ed13193faeb1 in term 1.
I20260810 03:40:29.549975 29497 leader_election.cc:302] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 610504ba740c46c88004c6b8611b083e, e889ae8ffce6406d80d7ed13193faeb1; no voters: 69c0ef9dc48b4bb5a29c2ca3b4eb44ca
I20260810 03:40:29.550096 29300 leader_election.cc:302] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 610504ba740c46c88004c6b8611b083e, e889ae8ffce6406d80d7ed13193faeb1; no voters: 69c0ef9dc48b4bb5a29c2ca3b4eb44ca
I20260810 03:40:29.550159 30070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 03:40:29.550210 30070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69c0ef9dc48b4bb5a29c2ca3b4eb44ca)
I20260810 03:40:29.550262 30070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:40:29.550532 30071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 03:40:29.550618 30071 raft_consensus.cc:482] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69c0ef9dc48b4bb5a29c2ca3b4eb44ca)
I20260810 03:40:29.550673 30071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:40:29.551537 30070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } }
I20260810 03:40:29.551932 30070 leader_election.cc:288] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [CANDIDATE]: Term 2 election: Requested vote from peers 69c0ef9dc48b4bb5a29c2ca3b4eb44ca (127.21.175.126:36825), e889ae8ffce6406d80d7ed13193faeb1 (127.21.175.125:43497)
I20260810 03:40:29.551899 30071 raft_consensus.cc:504] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } }
I20260810 03:40:29.552248 29327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "610504ba740c46c88004c6b8611b083e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e889ae8ffce6406d80d7ed13193faeb1"
I20260810 03:40:29.552449 29327 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 610504ba740c46c88004c6b8611b083e in current term 2: Already voted for candidate e889ae8ffce6406d80d7ed13193faeb1 in this term.
I20260810 03:40:29.552624 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:29.552388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:59164)
0810 03:40:29.552484 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:29.552540 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 03:40:29.552652 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:29.552072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:59150)
0810 03:40:29.552544 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:29.552590 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260810 03:40:29.552687 29523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e889ae8ffce6406d80d7ed13193faeb1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "610504ba740c46c88004c6b8611b083e"
I20260810 03:40:29.552855 29523 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e889ae8ffce6406d80d7ed13193faeb1 in current term 2: Already voted for candidate 610504ba740c46c88004c6b8611b083e in this term.
I20260810 03:40:29.552301 30071 leader_election.cc:288] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [CANDIDATE]: Term 2 election: Requested vote from peers 69c0ef9dc48b4bb5a29c2ca3b4eb44ca (127.21.175.126:36825), 610504ba740c46c88004c6b8611b083e (127.21.175.124:44501)
I20260810 03:40:29.553282 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:29.553047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:59176)
0810 03:40:29.553138 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:29.553195 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 03:40:29.553390 29302 leader_election.cc:334] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 69c0ef9dc48b4bb5a29c2ca3b4eb44ca (127.21.175.126:36825): Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
I20260810 03:40:29.553471 29302 leader_election.cc:302] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e889ae8ffce6406d80d7ed13193faeb1; no voters: 610504ba740c46c88004c6b8611b083e, 69c0ef9dc48b4bb5a29c2ca3b4eb44ca
I20260810 03:40:29.553640 30071 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 03:40:29.553967 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:29.553432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:59188)
0810 03:40:29.553875 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:29.553916 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":316}
W20260810 03:40:29.554093 29499 leader_election.cc:334] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 69c0ef9dc48b4bb5a29c2ca3b4eb44ca (127.21.175.126:36825): Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
I20260810 03:40:29.554184 29499 leader_election.cc:302] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610504ba740c46c88004c6b8611b083e; no voters: 69c0ef9dc48b4bb5a29c2ca3b4eb44ca, e889ae8ffce6406d80d7ed13193faeb1
I20260810 03:40:29.554318 30070 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 03:40:30.029958 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:30.029415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.67:53363)
0810 03:40:30.029750 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:30.029830 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":135,"threads_started":1}
I20260810 03:40:30.034569 30082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:30.034032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.65:58487)
0810 03:40:30.034370 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:30.034448 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":120,"threads_started":1}
I20260810 03:40:30.035636 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:30.035180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.66:45509)
0810 03:40:30.035477 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:30.035547 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":87,"threads_started":1}
I20260810 03:40:31.031683 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.031046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.67:47421)
0810 03:40:31.031441 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.031522 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":93,"threads_started":1}
I20260810 03:40:31.036319 30085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.035779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.65:49063)
0810 03:40:31.036104 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.036184 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":105,"threads_started":1}
I20260810 03:40:31.036816 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.036393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.66:60465)
0810 03:40:31.036662 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.036727 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":73,"threads_started":1}
I20260810 03:40:31.254863 30087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:31.255015 30087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } }
I20260810 03:40:31.255628 30087 leader_election.cc:288] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69c0ef9dc48b4bb5a29c2ca3b4eb44ca (127.21.175.126:36825), e889ae8ffce6406d80d7ed13193faeb1 (127.21.175.125:43497)
I20260810 03:40:31.256136 29327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "610504ba740c46c88004c6b8611b083e" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e889ae8ffce6406d80d7ed13193faeb1" is_pre_election: true
I20260810 03:40:31.256263 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.255843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:59436)
0810 03:40:31.256090 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.256151 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260810 03:40:31.256307 29327 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 610504ba740c46c88004c6b8611b083e in term 2.
I20260810 03:40:31.256718 29497 leader_election.cc:302] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 610504ba740c46c88004c6b8611b083e, e889ae8ffce6406d80d7ed13193faeb1; no voters:
I20260810 03:40:31.256939 30087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 03:40:31.257038 30087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:40:31.257105 30087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 2 FOLLOWER]: Advancing to term 3
I20260810 03:40:31.257166 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.256876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:59452)
0810 03:40:31.256975 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.257023 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 03:40:31.257287 29499 leader_election.cc:334] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 69c0ef9dc48b4bb5a29c2ca3b4eb44ca (127.21.175.126:36825): Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
I20260810 03:40:31.258304 30087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } }
I20260810 03:40:31.258657 30087 leader_election.cc:288] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [CANDIDATE]: Term 3 election: Requested vote from peers 69c0ef9dc48b4bb5a29c2ca3b4eb44ca (127.21.175.126:36825), e889ae8ffce6406d80d7ed13193faeb1 (127.21.175.125:43497)
I20260810 03:40:31.259030 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.258829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:59454)
0810 03:40:31.258922 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.258971 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 03:40:31.259057 29327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "610504ba740c46c88004c6b8611b083e" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e889ae8ffce6406d80d7ed13193faeb1"
I20260810 03:40:31.259217 29327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 2 FOLLOWER]: Advancing to term 3
I20260810 03:40:31.259672 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.259487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:59458)
0810 03:40:31.259581 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.259625 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 03:40:31.259780 29499 leader_election.cc:334] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 69c0ef9dc48b4bb5a29c2ca3b4eb44ca (127.21.175.126:36825): Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
I20260810 03:40:31.260462 29327 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 610504ba740c46c88004c6b8611b083e in term 3.
I20260810 03:40:31.260787 29497 leader_election.cc:302] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 610504ba740c46c88004c6b8611b083e, e889ae8ffce6406d80d7ed13193faeb1; no voters: 69c0ef9dc48b4bb5a29c2ca3b4eb44ca
I20260810 03:40:31.260995 30087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 3 FOLLOWER]: Leader election won for term 3
I20260810 03:40:31.261253 30087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [term 3 LEADER]: Becoming Leader. State: Replica: 610504ba740c46c88004c6b8611b083e, State: Running, Role: LEADER
I20260810 03:40:31.261607 30087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } }
I20260810 03:40:31.263082 30091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 610504ba740c46c88004c6b8611b083e. Latest consensus state: current_term: 3 leader_uuid: "610504ba740c46c88004c6b8611b083e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } } }
I20260810 03:40:31.263208 30091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:31.263540 30093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:31.266199 30093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:31.266647 30093 catalog_manager.cc:1259] Loaded cluster ID: 523daf2d91334e3999eb5540a2aa6f8c
I20260810 03:40:31.266695 30093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:31.267266 30093 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:31.267669 30093 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: Loaded TSK: 0
I20260810 03:40:31.268391 30093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:40:31.313242 29513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:40:31.316735 29513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:40:31.320188 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.319914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:59472)
0810 03:40:31.320050 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.320115 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 03:40:31.320344 29327 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [term 3 FOLLOWER]: Refusing update from remote peer 610504ba740c46c88004c6b8611b083e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260810 03:40:31.321065 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.320823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.0.0.1:59486)
0810 03:40:31.320952 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.321010 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 03:40:31.320971 30091 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260810 03:40:31.321213 29499 consensus_peers.cc:487] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e -> Peer 69c0ef9dc48b4bb5a29c2ca3b4eb44ca (127.21.175.126:36825): Couldn't send request to peer 69c0ef9dc48b4bb5a29c2ca3b4eb44ca. Status: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 03:40:31.322687 30095 sys_catalog.cc:455] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 610504ba740c46c88004c6b8611b083e. Latest consensus state: current_term: 3 leader_uuid: "610504ba740c46c88004c6b8611b083e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } } }
I20260810 03:40:31.322839 30095 sys_catalog.cc:458] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:40:31.324352 30095 sys_catalog.cc:455] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "610504ba740c46c88004c6b8611b083e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } } }
I20260810 03:40:31.324483 30095 sys_catalog.cc:458] T 00000000000000000000000000000000 P e889ae8ffce6406d80d7ed13193faeb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:40:31.324819 30091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "610504ba740c46c88004c6b8611b083e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "69c0ef9dc48b4bb5a29c2ca3b4eb44ca" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36825 } } peers { permanent_uuid: "e889ae8ffce6406d80d7ed13193faeb1" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 43497 } attrs { promote: false } } peers { permanent_uuid: "610504ba740c46c88004c6b8611b083e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 44501 } attrs { promote: false } } }
I20260810 03:40:31.324944 30091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:31.336632 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.331490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:34913 (local address 127.0.0.1:34212)
0810 03:40:31.331677 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.331693 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:31.331791 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:31.332668 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:31.332674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:31.332692 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:31.332807 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:31.332813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.333984 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.333988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:31.335247 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:31.335258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.336165 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.336169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:31.336273 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:31.336299 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:40:31.336388 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 03:40:31.336463 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":3}
I20260810 03:40:31.337138 30104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.332074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35932 (local address 127.21.175.65:37347)
0810 03:40:31.332833 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:31.332841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:31.332858 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:40:31.332913 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:31.332915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:31.332958 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:31.333041 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:31.333365 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.334168 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.335683 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.336464 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.336581 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:31.336664 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:31.336912 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 03:40:31.337008 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":105,"threads_started":1}
I20260810 03:40:31.337172 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.331819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.21.175.66:34913)
0810 03:40:31.332382 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:31.332387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:31.332403 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:40:31.332461 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:31.332466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:31.332527 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:31.332918 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:31.333048 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.333851 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.335431 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.336011 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.336156 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:31.336224 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:31.336943 (+ 719us) server_negotiation.cc:299] Negotiation successful
0810 03:40:31.337040 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":123,"threads_started":1}
I20260810 03:40:31.337567 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.331742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:37347 (local address 127.0.0.1:35932)
0810 03:40:31.332525 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.332538 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:31.332626 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:31.333062 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:31.333068 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:31.333093 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:31.333233 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:31.333242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.334364 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.334367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:31.335549 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:31.335557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.336598 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.336601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:31.336702 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:31.336733 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 03:40:31.336811 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 03:40:31.337350 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"spinlock_wait_cycles":5632,"thread_start_us":100,"threads_started":1}
I20260810 03:40:31.337705 29780 tablet_service.cc:1467] Processing CreateTablet for tablet d55b13aafbab4475b94b6e210e57ab9b (DEFAULT_TABLE table=test-table [id=fd42e990cf184e39bd4db135fc4499c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:31.338029 30105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.332567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:42793 (local address 127.0.0.1:41064)
0810 03:40:31.333019 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.333033 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:31.333131 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:31.333468 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:31.333471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:31.333519 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:31.333629 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:31.333641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.334848 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.334852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:31.336321 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:31.336333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.337519 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.337524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:31.337637 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:31.337674 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 03:40:31.337757 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 03:40:31.337844 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"spinlock_wait_cycles":384,"thread_start_us":105,"threads_started":1}
I20260810 03:40:31.338037 29780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55b13aafbab4475b94b6e210e57ab9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:31.338469 30106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.332690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41064 (local address 127.21.175.67:42793)
0810 03:40:31.332963 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:31.332969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:31.333221 (+ 252us) server_negotiation.cc:407] Connection header received
0810 03:40:31.333287 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:31.333293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:31.333356 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:31.333447 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:31.333828 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.334712 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.336483 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.337225 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.337356 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:31.337421 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:31.338228 (+ 807us) server_negotiation.cc:299] Negotiation successful
0810 03:40:31.338333 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":127,"threads_started":1}
I20260810 03:40:31.339319 29927 tablet_service.cc:1467] Processing CreateTablet for tablet d55b13aafbab4475b94b6e210e57ab9b (DEFAULT_TABLE table=test-table [id=fd42e990cf184e39bd4db135fc4499c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:31.339716 29927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55b13aafbab4475b94b6e210e57ab9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:31.340446 30107 tablet_bootstrap.cc:492] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c: Bootstrap starting.
I20260810 03:40:31.341641 30107 tablet_bootstrap.cc:654] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:31.342612 30110 tablet_bootstrap.cc:492] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438: Bootstrap starting.
I20260810 03:40:31.342875 30107 tablet_bootstrap.cc:492] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c: No bootstrap required, opened a new log
I20260810 03:40:31.342988 30107 ts_tablet_manager.cc:1397] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 03:40:31.343442 30107 raft_consensus.cc:348] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } }
I20260810 03:40:31.343602 30110 tablet_bootstrap.cc:654] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:31.343624 30107 raft_consensus.cc:374] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:31.343684 30107 raft_consensus.cc:729] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dda5f21ab0d4dd4bdef3d73df6bd04c, State: Initialized, Role: FOLLOWER
I20260810 03:40:31.343663 29633 tablet_service.cc:1467] Processing CreateTablet for tablet d55b13aafbab4475b94b6e210e57ab9b (DEFAULT_TABLE table=test-table [id=fd42e990cf184e39bd4db135fc4499c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:31.343844 30107 consensus_queue.cc:260] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } }
I20260810 03:40:31.343969 29633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55b13aafbab4475b94b6e210e57ab9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:31.344276 30107 ts_tablet_manager.cc:1428] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 03:40:31.344995 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.344690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.66:45023)
0810 03:40:31.344845 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.344918 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 03:40:31.345088 30110 tablet_bootstrap.cc:492] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438: No bootstrap required, opened a new log
I20260810 03:40:31.345170 30110 ts_tablet_manager.cc:1397] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:31.345691 30110 raft_consensus.cc:348] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } }
I20260810 03:40:31.345799 30110 raft_consensus.cc:374] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:31.345830 30110 raft_consensus.cc:729] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c144606813640a981f6128257243438, State: Initialized, Role: FOLLOWER
I20260810 03:40:31.345966 30110 consensus_queue.cc:260] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } }
I20260810 03:40:31.346390 30110 ts_tablet_manager.cc:1428] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 03:40:31.346971 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.346661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.67:48955)
0810 03:40:31.346828 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.346889 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260810 03:40:31.348601 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.345516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.66:59867)
0810 03:40:31.348493 (+ 2977us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.348541 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2915}
I20260810 03:40:31.349313 29997 heartbeater.cc:503] Master 127.21.175.124:44501 was elected leader, sending a full tablet report...
I20260810 03:40:31.349578 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.347456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.67:44947)
0810 03:40:31.349414 (+ 1958us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.349473 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1881}
I20260810 03:40:31.350442 30114 tablet_bootstrap.cc:492] T d55b13aafbab4475b94b6e210e57ab9b P 1061eeaa37a74bac8059dae22ab9da04: Bootstrap starting.
I20260810 03:40:31.351630 30114 tablet_bootstrap.cc:654] T d55b13aafbab4475b94b6e210e57ab9b P 1061eeaa37a74bac8059dae22ab9da04: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:31.353045 30114 tablet_bootstrap.cc:492] T d55b13aafbab4475b94b6e210e57ab9b P 1061eeaa37a74bac8059dae22ab9da04: No bootstrap required, opened a new log
I20260810 03:40:31.353174 30114 ts_tablet_manager.cc:1397] T d55b13aafbab4475b94b6e210e57ab9b P 1061eeaa37a74bac8059dae22ab9da04: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:31.353824 30114 raft_consensus.cc:348] T d55b13aafbab4475b94b6e210e57ab9b P 1061eeaa37a74bac8059dae22ab9da04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } }
I20260810 03:40:31.353993 30114 raft_consensus.cc:374] T d55b13aafbab4475b94b6e210e57ab9b P 1061eeaa37a74bac8059dae22ab9da04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:31.354076 30114 raft_consensus.cc:729] T d55b13aafbab4475b94b6e210e57ab9b P 1061eeaa37a74bac8059dae22ab9da04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1061eeaa37a74bac8059dae22ab9da04, State: Initialized, Role: FOLLOWER
I20260810 03:40:31.354234 30114 consensus_queue.cc:260] T d55b13aafbab4475b94b6e210e57ab9b P 1061eeaa37a74bac8059dae22ab9da04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } }
I20260810 03:40:31.354622 30114 ts_tablet_manager.cc:1428] T d55b13aafbab4475b94b6e210e57ab9b P 1061eeaa37a74bac8059dae22ab9da04: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 03:40:31.355896 30085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.355297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.65:36739)
0810 03:40:31.355715 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.355786 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336}
I20260810 03:40:31.356499 29706 heartbeater.cc:503] Master 127.21.175.124:44501 was elected leader, sending a full tablet report...
I20260810 03:40:31.357342 30085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.357049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.65:39623)
0810 03:40:31.357209 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.357267 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 03:40:31.466264 30113 raft_consensus.cc:482] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:31.466589 30113 raft_consensus.cc:504] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } }
I20260810 03:40:31.467854 30113 leader_election.cc:288] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dda5f21ab0d4dd4bdef3d73df6bd04c (127.21.175.66:34913), 1061eeaa37a74bac8059dae22ab9da04 (127.21.175.65:37347)
I20260810 03:40:31.472208 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.468462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:58133 (local address 127.21.175.66:34913)
0810 03:40:31.468623 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:31.468628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:31.469045 (+ 417us) server_negotiation.cc:407] Connection header received
0810 03:40:31.469197 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:31.469200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:31.469250 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:31.469337 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:31.469682 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.470190 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.471217 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.471619 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.471758 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:31.471817 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:31.472008 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 03:40:31.472091 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:40:31.472234 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.468262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:34913 (local address 127.21.175.67:58133)
0810 03:40:31.468936 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.468948 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:31.469076 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:31.469350 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:31.469353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:31.469369 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:31.469514 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:31.469520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.470319 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.470322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:31.471075 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:31.471084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.471764 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.471768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:31.471870 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:31.471900 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 03:40:31.471989 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 03:40:31.472063 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":295,"thread_start_us":154,"threads_started":1}
I20260810 03:40:31.472275 30104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.468545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:39823 (local address 127.21.175.65:37347)
0810 03:40:31.468706 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:31.468712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:31.468730 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:40:31.468789 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:31.468791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:31.468838 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:31.468950 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:31.469278 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.469919 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.471215 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.471654 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.471801 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:31.471860 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:31.472074 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 03:40:31.472157 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:40:31.472275 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.468262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:37347 (local address 127.21.175.67:39823)
0810 03:40:31.468450 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.468469 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:31.468608 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:31.468979 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:31.468982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:31.469004 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:31.469147 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:31.469153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.470048 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.470054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:31.471021 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:31.471032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.471796 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.471800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:31.471915 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:31.471952 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 03:40:31.472056 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 03:40:31.472133 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 03:40:31.472827 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55b13aafbab4475b94b6e210e57ab9b" candidate_uuid: "9c144606813640a981f6128257243438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1061eeaa37a74bac8059dae22ab9da04" is_pre_election: true
I20260810 03:40:31.472852 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55b13aafbab4475b94b6e210e57ab9b" candidate_uuid: "9c144606813640a981f6128257243438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" is_pre_election: true
I20260810 03:40:31.473008 29653 raft_consensus.cc:2454] T d55b13aafbab4475b94b6e210e57ab9b P 1061eeaa37a74bac8059dae22ab9da04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c144606813640a981f6128257243438 in term 0.
I20260810 03:40:31.473080 29801 raft_consensus.cc:2454] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c144606813640a981f6128257243438 in term 0.
I20260810 03:40:31.473500 29877 leader_election.cc:302] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1061eeaa37a74bac8059dae22ab9da04, 9c144606813640a981f6128257243438; no voters:
I20260810 03:40:31.473834 30113 raft_consensus.cc:2781] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:40:31.473898 30113 raft_consensus.cc:482] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:40:31.473928 30113 raft_consensus.cc:3037] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:31.475023 30113 raft_consensus.cc:504] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } }
I20260810 03:40:31.475450 30113 leader_election.cc:288] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [CANDIDATE]: Term 1 election: Requested vote from peers 0dda5f21ab0d4dd4bdef3d73df6bd04c (127.21.175.66:34913), 1061eeaa37a74bac8059dae22ab9da04 (127.21.175.65:37347)
I20260810 03:40:31.475847 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55b13aafbab4475b94b6e210e57ab9b" candidate_uuid: "9c144606813640a981f6128257243438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c"
I20260810 03:40:31.475925 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55b13aafbab4475b94b6e210e57ab9b" candidate_uuid: "9c144606813640a981f6128257243438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1061eeaa37a74bac8059dae22ab9da04"
I20260810 03:40:31.475983 29801 raft_consensus.cc:3037] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:31.476034 29653 raft_consensus.cc:3037] T d55b13aafbab4475b94b6e210e57ab9b P 1061eeaa37a74bac8059dae22ab9da04 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:31.477222 29801 raft_consensus.cc:2454] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c144606813640a981f6128257243438 in term 1.
I20260810 03:40:31.477233 29653 raft_consensus.cc:2454] T d55b13aafbab4475b94b6e210e57ab9b P 1061eeaa37a74bac8059dae22ab9da04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c144606813640a981f6128257243438 in term 1.
I20260810 03:40:31.477586 29877 leader_election.cc:302] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1061eeaa37a74bac8059dae22ab9da04, 9c144606813640a981f6128257243438; no voters:
I20260810 03:40:31.477758 30113 raft_consensus.cc:2781] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:31.478018 30113 raft_consensus.cc:686] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [term 1 LEADER]: Becoming Leader. State: Replica: 9c144606813640a981f6128257243438, State: Running, Role: LEADER
I20260810 03:40:31.478320 30113 consensus_queue.cc:237] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } } peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } }
I20260810 03:40:31.478816 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.478573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.67:57511)
0810 03:40:31.478685 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.478751 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 03:40:31.480736 29513 catalog_manager.cc:5495] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c144606813640a981f6128257243438 (127.21.175.67). New cstate: current_term: 1 leader_uuid: "9c144606813640a981f6128257243438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c144606813640a981f6128257243438" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42793 } health_report { overall_health: HEALTHY } } }
I20260810 03:40:31.495157 30106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.491494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.21.175.67:42793)
0810 03:40:31.491640 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:31.491644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:31.491678 (+ 34us) server_negotiation.cc:407] Connection header received
0810 03:40:31.491756 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:31.491758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:31.491815 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:31.491901 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:40:31.492123 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.492696 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.493633 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:31.493842 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:31.493919 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:31.494174 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:40:31.494709 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:40:31.494802 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:31.494914 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 03:40:31.494971 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 03:40:31.498574 29653 raft_consensus.cc:1261] T d55b13aafbab4475b94b6e210e57ab9b P 1061eeaa37a74bac8059dae22ab9da04 [term 1 FOLLOWER]: Refusing update from remote peer 9c144606813640a981f6128257243438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:31.498661 29801 raft_consensus.cc:1261] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 1 FOLLOWER]: Refusing update from remote peer 9c144606813640a981f6128257243438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:31.499222 30113 consensus_queue.cc:1035] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1061eeaa37a74bac8059dae22ab9da04" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:31.499490 30119 consensus_queue.cc:1035] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dda5f21ab0d4dd4bdef3d73df6bd04c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 34913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:31.501025 29853 heartbeater.cc:503] Master 127.21.175.124:44501 was elected leader, sending a full tablet report...
I20260810 03:40:31.501536 30085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.501167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.65:49727)
0810 03:40:31.501358 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.501425 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260810 03:40:31.502251 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.501936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.66:55687)
0810 03:40:31.502085 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.502145 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260810 03:40:31.504379 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.504054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.67:37859)
0810 03:40:31.504226 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.504284 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 03:40:31.506098 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.505788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.66:52065)
0810 03:40:31.505957 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.506009 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 03:40:31.507145 30125 mvcc.cc:205] Tried to move back new op lower bound from 7316820916211580928 to 7316820916138045440. Current Snapshot: MvccSnapshot[applied={T|T < 7316820916211580928 or (T in {7316820916211580928})}]
I20260810 03:40:31.508234 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.507697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.67:46611)
0810 03:40:31.508093 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.508155 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310}
I20260810 03:40:31.508852 30085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.508501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36825 (local address 127.21.175.65:58921)
0810 03:40:31.508716 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.508775 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260810 03:40:31.511464 30124 mvcc.cc:205] Tried to move back new op lower bound from 7316820916211580928 to 7316820916138045440. Current Snapshot: MvccSnapshot[applied={T|T < 7316820916211580928}]
I20260810 03:40:31.517946 29512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:31.518181 29512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:31.522934 29512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 610504ba740c46c88004c6b8611b083e: Sending DeleteTablet for 3 replicas of tablet d55b13aafbab4475b94b6e210e57ab9b
I20260810 03:40:31.523821 29927 tablet_service.cc:1514] Processing DeleteTablet for tablet d55b13aafbab4475b94b6e210e57ab9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:31 UTC) from {username='oryx', principal='oryx/127.21.175.124@KRBTEST.COM'} at 127.0.0.1:41064
I20260810 03:40:31.523851 29633 tablet_service.cc:1514] Processing DeleteTablet for tablet d55b13aafbab4475b94b6e210e57ab9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:31 UTC) from {username='oryx', principal='oryx/127.21.175.124@KRBTEST.COM'} at 127.0.0.1:35932
I20260810 03:40:31.524024 29780 tablet_service.cc:1514] Processing DeleteTablet for tablet d55b13aafbab4475b94b6e210e57ab9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:31 UTC) from {username='oryx', principal='oryx/127.21.175.124@KRBTEST.COM'} at 127.0.0.1:34212
I20260810 03:40:31.524300 30133 tablet_replica.cc:330] T d55b13aafbab4475b94b6e210e57ab9b P 1061eeaa37a74bac8059dae22ab9da04: stopping tablet replica
I20260810 03:40:31.524428 30133 raft_consensus.cc:2229] T d55b13aafbab4475b94b6e210e57ab9b P 1061eeaa37a74bac8059dae22ab9da04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:31.524587 30133 raft_consensus.cc:2258] T d55b13aafbab4475b94b6e210e57ab9b P 1061eeaa37a74bac8059dae22ab9da04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:31.524631 30134 tablet_replica.cc:330] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c: stopping tablet replica
I20260810 03:40:31.524803 30134 raft_consensus.cc:2229] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:31.524863 30132 tablet_replica.cc:330] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438: stopping tablet replica
I20260810 03:40:31.524999 30134 raft_consensus.cc:2258] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:31.525043 30132 raft_consensus.cc:2229] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:40:31.525511 30132 raft_consensus.cc:2258] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:31.526136 30134 ts_tablet_manager.cc:1905] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:31.526278 30133 ts_tablet_manager.cc:1905] T d55b13aafbab4475b94b6e210e57ab9b P 1061eeaa37a74bac8059dae22ab9da04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:31.526758 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 29469
I20260810 03:40:31.530781 30134 ts_tablet_manager.cc:1918] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:40:31.530891 30134 log.cc:1196] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/d55b13aafbab4475b94b6e210e57ab9b
I20260810 03:40:31.531216 30134 ts_tablet_manager.cc:1939] T d55b13aafbab4475b94b6e210e57ab9b P 0dda5f21ab0d4dd4bdef3d73df6bd04c: Deleting consensus metadata
I20260810 03:40:31.532460 29499 catalog_manager.cc:4841] TS 0dda5f21ab0d4dd4bdef3d73df6bd04c (127.21.175.66:34913): tablet d55b13aafbab4475b94b6e210e57ab9b (table test-table [id=fd42e990cf184e39bd4db135fc4499c7]) successfully deleted
I20260810 03:40:31.534235 30132 ts_tablet_manager.cc:1905] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:31.536983 30132 ts_tablet_manager.cc:1918] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:40:31.537177 30132 log.cc:1196] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/d55b13aafbab4475b94b6e210e57ab9b
I20260810 03:40:31.537580 30132 ts_tablet_manager.cc:1939] T d55b13aafbab4475b94b6e210e57ab9b P 9c144606813640a981f6128257243438: Deleting consensus metadata
I20260810 03:40:31.537842 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 29714
I20260810 03:40:31.537990 30105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:31.537600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:37347 (local address 127.0.0.1:35940)
0810 03:40:31.537819 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:31.537881 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260810 03:40:31.540050 29496 catalog_manager.cc:4584] TS 1061eeaa37a74bac8059dae22ab9da04 (127.21.175.65:37347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d55b13aafbab4475b94b6e210e57ab9b: Network error: Client connection negotiation failed: client connection to 127.21.175.65:37347: connect: Connection refused (error 111)
I20260810 03:40:31.541055 29498 catalog_manager.cc:4841] TS 9c144606813640a981f6128257243438 (127.21.175.67:42793): tablet d55b13aafbab4475b94b6e210e57ab9b (table test-table [id=fd42e990cf184e39bd4db135fc4499c7]) successfully deleted
I20260810 03:40:31.548043 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 29859
I20260810 03:40:31.557766 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 29171
I20260810 03:40:31.566589 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 29279
2026-08-10T03:40:31Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7533 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:40:31 dist-test-slave-ss4n krb5kdc[30137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:40:31 dist-test-slave-ss4n krb5kdc[30137](info): set up 2 sockets
Aug 10 03:40:31 dist-test-slave-ss4n krb5kdc[30137](info): commencing operation
krb5kdc: starting...
W20260810 03:40:32.715049 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:40:32 dist-test-slave-ss4n krb5kdc[30137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:40:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.126.keytab.
Entry for principal oryx/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.126.keytab.
I20260810 03:40:32.857328 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:36197
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.126.keytab
--principal=oryx/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:36197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:32.976913 30150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:32.977262 30150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:32.977344 30150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:32.981576 30150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:32.981678 30150 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 03:40:32.981707 30150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:32.981750 30150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:32.981784 30150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:32.981817 30150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:40:32.987648 30150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:36197
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.126.keytab
--principal=oryx/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:36197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:32.988839 30150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:32.989987 30150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:32.999785 30155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:32.999785 30156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:32.999785 30158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:33.000138 30150 server_base.cc:1034] running on GCE node
I20260810 03:40:33.000671 30150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:33.001914 30150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:33.003095 30150 hybrid_clock.cc:648] HybridClock initialized: now 1786333233003062 us; error 47 us; skew 500 ppm
Aug 10 03:40:33 dist-test-slave-ss4n krb5kdc[30137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333233, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:33.006139 30150 init.cc:377] Logged in from keytab as oryx/127.21.175.126@KRBTEST.COM (short username oryx)
I20260810 03:40:33.007119 30150 webserver.cc:460] Webserver started at http://127.21.175.126:46603/ using document root <none> and password file <none>
I20260810 03:40:33.007706 30150 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:33.007802 30150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:33.008054 30150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:33.010383 30150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "d4e77b4104c04dadaeea97b78db5924c"
format_stamp: "Formatted at 2026-08-10 03:40:33 on dist-test-slave-ss4n"
server_key: "9b7051fea03f00a27400dd7b751ab514"
server_key_iv: "48a25d8fb111a8d0976f7e9f3f6391cc"
server_key_version: "encryptionkey@0"
I20260810 03:40:33.011056 30150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "d4e77b4104c04dadaeea97b78db5924c"
format_stamp: "Formatted at 2026-08-10 03:40:33 on dist-test-slave-ss4n"
server_key: "9b7051fea03f00a27400dd7b751ab514"
server_key_iv: "48a25d8fb111a8d0976f7e9f3f6391cc"
server_key_version: "encryptionkey@0"
I20260810 03:40:33.015239 30150 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:40:33.018077 30165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:33.019387 30150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:33.019546 30150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "d4e77b4104c04dadaeea97b78db5924c"
format_stamp: "Formatted at 2026-08-10 03:40:33 on dist-test-slave-ss4n"
server_key: "9b7051fea03f00a27400dd7b751ab514"
server_key_iv: "48a25d8fb111a8d0976f7e9f3f6391cc"
server_key_version: "encryptionkey@0"
I20260810 03:40:33.019675 30150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:33.036502 30150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:33.040550 30150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:33.040812 30150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:33.050616 30150 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:36197
I20260810 03:40:33.050653 30231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:36197 every 8 connection(s)
I20260810 03:40:33.051718 30150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:33.054785 30232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:33.055629 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 30150
I20260810 03:40:33.055737 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:40:33.056085 22205 external_mini_cluster.cc:1428] Setting key b15a7bd48a152a885e2af7515f309f3e
I20260810 03:40:33.062230 30232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c: Bootstrap starting.
I20260810 03:40:33.065140 30232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:33.066067 30232 log.cc:824] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:40:33 dist-test-slave-ss4n krb5kdc[30137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM
I20260810 03:40:33.068493 30232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c: No bootstrap required, opened a new log
I20260810 03:40:33.071847 30232 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e77b4104c04dadaeea97b78db5924c" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36197 } }
I20260810 03:40:33.072120 30232 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:33.072180 30232 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e77b4104c04dadaeea97b78db5924c, State: Initialized, Role: FOLLOWER
I20260810 03:40:33.072678 30232 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e77b4104c04dadaeea97b78db5924c" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36197 } }
I20260810 03:40:33.072852 30232 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:33.072938 30232 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:33.073062 30232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:33.073411 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.057658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.21.175.126:36197)
0810 03:40:33.058137 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:33.058158 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:33.058205 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:40:33.059042 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:33.059079 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:33.059533 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:33.059972 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:33.060687 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.062253 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.063196 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.063951 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.064128 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.067351 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:33.067357 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:33.067373 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:33.067403 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:33.070066 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:33.070657 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:33.070662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:33.070668 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:33.070803 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:33.071125 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:33.071130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:33.071131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:33.071573 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:33.071798 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:33.072146 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 03:40:33.072492 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":122,"threads_started":1}
I20260810 03:40:33.074347 30232 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e77b4104c04dadaeea97b78db5924c" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36197 } }
I20260810 03:40:33.074725 30232 leader_election.cc:302] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4e77b4104c04dadaeea97b78db5924c; no voters:
I20260810 03:40:33.075059 30232 leader_election.cc:288] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:40:33.075182 30237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:33.075410 30237 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [term 1 LEADER]: Becoming Leader. State: Replica: d4e77b4104c04dadaeea97b78db5924c, State: Running, Role: LEADER
I20260810 03:40:33.075733 30237 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e77b4104c04dadaeea97b78db5924c" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36197 } }
I20260810 03:40:33.076478 30232 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:33.077644 30239 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4e77b4104c04dadaeea97b78db5924c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e77b4104c04dadaeea97b78db5924c" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36197 } } }
I20260810 03:40:33.077776 30239 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:33.078199 30247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:33.078531 30238 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4e77b4104c04dadaeea97b78db5924c. Latest consensus state: current_term: 1 leader_uuid: "d4e77b4104c04dadaeea97b78db5924c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e77b4104c04dadaeea97b78db5924c" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36197 } } }
I20260810 03:40:33.078634 30238 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:33.082871 30247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:33.088783 30247 catalog_manager.cc:1347] Generated new cluster ID: c7a4cf6ea6b241f5a06760b93d2ca148
I20260810 03:40:33.088876 30247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:33.094058 30247 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:40:33.095460 30247 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:33.102497 30247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c: Generated new TSK 0
I20260810 03:40:33.103336 30247 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.65.keytab.
Entry for principal oryx/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.65.keytab.
I20260810 03:40:33.149422 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36197
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.65.keytab
--principal=oryx/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:33.273299 30259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:33.273701 30259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:33.273804 30259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:33.278432 30259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:33.278527 30259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:33.278607 30259 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:40:33.278697 30259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:40:33.284335 30259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.65.keytab
--principal=oryx/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.175.126:36197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:33.285635 30259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:33.286813 30259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:33.297168 30265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:33.297180 30266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:33.297346 30259 server_base.cc:1034] running on GCE node
W20260810 03:40:33.297196 30268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:33.298061 30259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:33.298806 30259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:33.300040 30259 hybrid_clock.cc:648] HybridClock initialized: now 1786333233300024 us; error 44 us; skew 500 ppm
Aug 10 03:40:33 dist-test-slave-ss4n krb5kdc[30137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333233, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:33.303417 30259 init.cc:377] Logged in from keytab as oryx/127.21.175.65@KRBTEST.COM (short username oryx)
I20260810 03:40:33.304539 30259 webserver.cc:460] Webserver started at http://127.21.175.65:37591/ using document root <none> and password file <none>
I20260810 03:40:33.305204 30259 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:33.305315 30259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:33.305660 30259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:33.308002 30259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "122fedfa36e54e0eb3a9ed06808551f3"
format_stamp: "Formatted at 2026-08-10 03:40:33 on dist-test-slave-ss4n"
server_key: "0312f265745a88aed3bb67bc8ccf8406"
server_key_iv: "c3d4f6a3820b9b7444870ee9f54a70f9"
server_key_version: "encryptionkey@0"
I20260810 03:40:33.308691 30259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "122fedfa36e54e0eb3a9ed06808551f3"
format_stamp: "Formatted at 2026-08-10 03:40:33 on dist-test-slave-ss4n"
server_key: "0312f265745a88aed3bb67bc8ccf8406"
server_key_iv: "c3d4f6a3820b9b7444870ee9f54a70f9"
server_key_version: "encryptionkey@0"
I20260810 03:40:33.313159 30259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:40:33.316293 30276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:33.317965 30259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 03:40:33.318130 30259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "122fedfa36e54e0eb3a9ed06808551f3"
format_stamp: "Formatted at 2026-08-10 03:40:33 on dist-test-slave-ss4n"
server_key: "0312f265745a88aed3bb67bc8ccf8406"
server_key_iv: "c3d4f6a3820b9b7444870ee9f54a70f9"
server_key_version: "encryptionkey@0"
I20260810 03:40:33.318272 30259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:33.329963 30259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:33.333830 30259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:33.334065 30259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:33.335611 30259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:33.335695 30259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:33.335768 30259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:33.335822 30259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:33.347441 30259 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:41379
Aug 10 03:40:33 dist-test-slave-ss4n krb5kdc[30137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333233, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM
I20260810 03:40:33.347590 30398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:41379 every 8 connection(s)
I20260810 03:40:33.348590 30259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:40:33.352886 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.336864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:46127 (local address 127.21.175.126:36197)
0810 03:40:33.337037 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:33.337041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:33.338293 (+ 1252us) server_negotiation.cc:407] Connection header received
0810 03:40:33.339833 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:33.339838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:33.339899 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:33.340003 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:33.342055 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.342998 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.344476 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.344878 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.345022 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.348408 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:33.348412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:33.348414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:33.348513 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:33.350489 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:33.351137 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:33.351140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:33.351142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:33.351233 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:33.351631 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:33.351634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:33.351635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:33.351841 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:33.351925 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:33.352620 (+ 695us) server_negotiation.cc:299] Negotiation successful
0810 03:40:33.352744 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:40:33.354004 30286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.337471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36197 (local address 127.21.175.65:46127)
0810 03:40:33.338096 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:33.338147 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:33.339538 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:33.340271 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:33.340282 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:33.340920 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:33.341800 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.341854 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.343182 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.343188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.344323 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.344334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.345110 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.345117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.345361 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.346169 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:33.346187 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:33.348170 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:33.350628 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:33.350633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:33.350647 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:33.350979 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:33.351325 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:33.351327 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:33.351329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:33.351471 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:33.352043 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:33.352049 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:33.352330 (+ 281us) client_negotiation.cc:769] Sending connection context
0810 03:40:33.352555 (+ 225us) client_negotiation.cc:240] Negotiation successful
0810 03:40:33.352880 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":152,"threads_started":1}
I20260810 03:40:33.355774 30399 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36197
I20260810 03:40:33.356005 30399 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:33.356688 30399 heartbeater.cc:511] Master 127.21.175.126:36197 requested a full tablet report, sending...
I20260810 03:40:33.358250 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 30259
I20260810 03:40:33.358397 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:40:33.358762 22205 external_mini_cluster.cc:1428] Setting key 2938d84f5e70a284f9914d96a6e5ae2c
I20260810 03:40:33.358996 30185 ts_manager.cc:194] Registered new tserver with Master: 122fedfa36e54e0eb3a9ed06808551f3 (127.21.175.65:41379)
I20260810 03:40:33.360693 30185 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:46127
I20260810 03:40:33.369987 30405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.358758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36197 (local address 127.21.175.65:37513)
0810 03:40:33.359431 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:33.359447 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:33.359557 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:33.359874 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:33.359877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:33.360141 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:33.360299 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.360306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.362694 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.362698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.363593 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.363606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.364169 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.364172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.364258 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.364986 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:33.365000 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:33.365397 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:33.367994 (+ 2597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:33.367997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:33.367999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:33.368248 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:33.368626 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:33.368628 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:33.368630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:33.368742 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:33.369269 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:33.369271 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:33.369381 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 03:40:33.369608 (+ 227us) client_negotiation.cc:240] Negotiation successful
0810 03:40:33.369774 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":138,"threads_started":1}
I20260810 03:40:33.370077 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.358821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:37513 (local address 127.21.175.126:36197)
0810 03:40:33.359263 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:33.359268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:33.359521 (+ 253us) server_negotiation.cc:407] Connection header received
0810 03:40:33.359665 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:33.359669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:33.359720 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:33.359822 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:33.361660 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.362517 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.363745 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.364024 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.364117 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.365639 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:33.365646 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:33.365652 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:33.365690 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:33.367833 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:33.368361 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:33.368365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:33.368367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:33.368471 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:33.368857 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:33.368861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:33.368863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:33.369139 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:33.369253 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:33.369701 (+ 448us) server_negotiation.cc:299] Negotiation successful
0810 03:40:33.369857 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":352}
WARNING: no policy specified for oryx/127.21.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.66.keytab.
Entry for principal oryx/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.66.keytab.
I20260810 03:40:33.394773 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36197
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.66.keytab
--principal=oryx/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:33.518321 30408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:33.518640 30408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:33.518688 30408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:33.523187 30408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:33.523283 30408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:33.523347 30408 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:40:33.523393 30408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:40:33.529217 30408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.66.keytab
--principal=oryx/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.175.126:36197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:33.530503 30408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:33.531607 30408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:33.542371 30413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:33.542366 30416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:33.542342 30414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:33.542635 30408 server_base.cc:1034] running on GCE node
I20260810 03:40:33.543152 30408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:33.544013 30408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:33.545217 30408 hybrid_clock.cc:648] HybridClock initialized: now 1786333233545201 us; error 30 us; skew 500 ppm
Aug 10 03:40:33 dist-test-slave-ss4n krb5kdc[30137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333233, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:33.548372 30408 init.cc:377] Logged in from keytab as oryx/127.21.175.66@KRBTEST.COM (short username oryx)
I20260810 03:40:33.549430 30408 webserver.cc:460] Webserver started at http://127.21.175.66:45229/ using document root <none> and password file <none>
I20260810 03:40:33.550065 30408 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:33.550163 30408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:33.550408 30408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:33.552593 30408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "4962c93336104d7790c2f3e5d1ed5c8c"
format_stamp: "Formatted at 2026-08-10 03:40:33 on dist-test-slave-ss4n"
server_key: "be50a1c9da72e08b169ead5a4aa431da"
server_key_iv: "fa71885a618e4d8749bee02c9836518d"
server_key_version: "encryptionkey@0"
I20260810 03:40:33.553226 30408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "4962c93336104d7790c2f3e5d1ed5c8c"
format_stamp: "Formatted at 2026-08-10 03:40:33 on dist-test-slave-ss4n"
server_key: "be50a1c9da72e08b169ead5a4aa431da"
server_key_iv: "fa71885a618e4d8749bee02c9836518d"
server_key_version: "encryptionkey@0"
I20260810 03:40:33.557633 30408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:40:33.560349 30424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:33.561825 30408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:33.561970 30408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "4962c93336104d7790c2f3e5d1ed5c8c"
format_stamp: "Formatted at 2026-08-10 03:40:33 on dist-test-slave-ss4n"
server_key: "be50a1c9da72e08b169ead5a4aa431da"
server_key_iv: "fa71885a618e4d8749bee02c9836518d"
server_key_version: "encryptionkey@0"
I20260810 03:40:33.562098 30408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:33.576145 30408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:33.580394 30408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:33.580646 30408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:33.582352 30408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:33.582441 30408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:33.582525 30408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:33.582587 30408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:40:33 dist-test-slave-ss4n krb5kdc[30137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333233, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM
I20260810 03:40:33.595013 30408 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:39665
I20260810 03:40:33.595289 30546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:39665 every 8 connection(s)
I20260810 03:40:33.596320 30408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:40:33.598196 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.583184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:38333 (local address 127.21.175.126:36197)
0810 03:40:33.583322 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:33.583331 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:33.584522 (+ 1191us) server_negotiation.cc:407] Connection header received
0810 03:40:33.585581 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:33.585584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:33.585645 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:33.585720 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:33.587307 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.588146 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.589930 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.590315 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.590435 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.594070 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:33.594074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:33.594077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:33.594128 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:33.595929 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:33.596478 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:33.596481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:33.596483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:33.596576 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:33.596979 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:33.596982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:33.596984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:33.597239 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:33.597333 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:33.597968 (+ 635us) server_negotiation.cc:299] Negotiation successful
0810 03:40:33.598077 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:40:33.599112 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.583822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36197 (local address 127.21.175.66:38333)
0810 03:40:33.584363 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:33.584408 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:33.585320 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:33.585898 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:33.585907 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:33.586427 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:33.587148 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.587168 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.588262 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.588268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.589820 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.589833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.590543 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.590551 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.590847 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.591639 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:33.591664 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:33.593887 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:33.596072 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:33.596078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:33.596094 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:33.596361 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:33.596686 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:33.596689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:33.596692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:33.596863 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:33.597370 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:33.597377 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:33.597729 (+ 352us) client_negotiation.cc:769] Sending connection context
0810 03:40:33.597899 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 03:40:33.598191 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":166,"threads_started":1}
I20260810 03:40:33.600380 30547 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36197
I20260810 03:40:33.600533 30547 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:33.601063 30547 heartbeater.cc:511] Master 127.21.175.126:36197 requested a full tablet report, sending...
I20260810 03:40:33.602236 30186 ts_manager.cc:194] Registered new tserver with Master: 4962c93336104d7790c2f3e5d1ed5c8c (127.21.175.66:39665)
I20260810 03:40:33.603052 30186 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:38333
I20260810 03:40:33.603907 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 30408
I20260810 03:40:33.604028 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:40:33.604341 22205 external_mini_cluster.cc:1428] Setting key 947a8be3f058caa13cb48770608e1bf0
I20260810 03:40:33.612043 30553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.603034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36197 (local address 127.21.175.66:46653)
0810 03:40:33.603407 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:33.603423 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:33.603519 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:33.603782 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:33.603786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:33.604061 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:33.604289 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.604298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.605357 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.605361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.606255 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.606270 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.606762 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.606768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.606899 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.607465 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:33.607485 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:33.607994 (+ 509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:33.610284 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:33.610287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:33.610288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:33.610486 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:33.610815 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:33.610817 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:33.610819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:33.610896 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:33.611475 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:33.611477 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:33.611558 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 03:40:33.611662 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 03:40:33.611838 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":24,"thread_start_us":115,"threads_started":1}
I20260810 03:40:33.612181 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.603110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:46653 (local address 127.21.175.126:36197)
0810 03:40:33.603237 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:33.603241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:33.603504 (+ 263us) server_negotiation.cc:407] Connection header received
0810 03:40:33.603616 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:33.603620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:33.603670 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:33.603766 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:33.604467 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.605213 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.606389 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.606637 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.606730 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.608167 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:33.608175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:33.608179 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:33.608214 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:33.610163 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:33.610594 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:33.610600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:33.610606 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:33.610714 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:33.610989 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:33.610995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:33.611000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:33.611365 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:33.611516 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:33.611768 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 03:40:33.611929 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for oryx/127.21.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.67.keytab.
Entry for principal oryx/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.67.keytab.
I20260810 03:40:33.640494 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36197
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.67.keytab
--principal=oryx/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:33.766358 30557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:33.766760 30557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:33.766857 30557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:33.771204 30557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:33.771276 30557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:33.771319 30557 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:40:33.771353 30557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:40:33.776921 30557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.67.keytab
--principal=oryx/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.175.126:36197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:33.778169 30557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:33.779281 30557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:33.789654 30563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:33.789605 30557 server_base.cc:1034] running on GCE node
W20260810 03:40:33.789535 30564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:33.789529 30566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:33.790308 30557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:33.791090 30557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:33.792294 30557 hybrid_clock.cc:648] HybridClock initialized: now 1786333233792267 us; error 36 us; skew 500 ppm
Aug 10 03:40:33 dist-test-slave-ss4n krb5kdc[30137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333233, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:33.796000 30557 init.cc:377] Logged in from keytab as oryx/127.21.175.67@KRBTEST.COM (short username oryx)
I20260810 03:40:33.797186 30557 webserver.cc:460] Webserver started at http://127.21.175.67:40391/ using document root <none> and password file <none>
I20260810 03:40:33.797943 30557 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:33.798025 30557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:33.798220 30557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:33.800549 30557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "00fe9b28af39403f8287d5ad827007e3"
format_stamp: "Formatted at 2026-08-10 03:40:33 on dist-test-slave-ss4n"
server_key: "01723c061bd0d9359e347006626d34fa"
server_key_iv: "cfb61fd886c4ded25e0b807294b12fb5"
server_key_version: "encryptionkey@0"
I20260810 03:40:33.801303 30557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "00fe9b28af39403f8287d5ad827007e3"
format_stamp: "Formatted at 2026-08-10 03:40:33 on dist-test-slave-ss4n"
server_key: "01723c061bd0d9359e347006626d34fa"
server_key_iv: "cfb61fd886c4ded25e0b807294b12fb5"
server_key_version: "encryptionkey@0"
I20260810 03:40:33.805837 30557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:40:33.808509 30573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:33.809988 30557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:33.810137 30557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "00fe9b28af39403f8287d5ad827007e3"
format_stamp: "Formatted at 2026-08-10 03:40:33 on dist-test-slave-ss4n"
server_key: "01723c061bd0d9359e347006626d34fa"
server_key_iv: "cfb61fd886c4ded25e0b807294b12fb5"
server_key_version: "encryptionkey@0"
I20260810 03:40:33.810250 30557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:33.825644 30557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:33.829224 30557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:33.829443 30557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:33.831115 30557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:33.831198 30557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:33.831260 30557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:33.831310 30557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:40:33 dist-test-slave-ss4n krb5kdc[30137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333233, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM
I20260810 03:40:33.843253 30557 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:39917
I20260810 03:40:33.843358 30697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:39917 every 8 connection(s)
I20260810 03:40:33.844434 30557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:40:33.846853 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.832207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:44151 (local address 127.21.175.126:36197)
0810 03:40:33.832355 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:33.832359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:33.833608 (+ 1249us) server_negotiation.cc:407] Connection header received
0810 03:40:33.834812 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:33.834816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:33.834879 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:33.834990 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:33.836533 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.837069 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.838435 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.838853 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.839004 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.842559 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:33.842562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:33.842565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:33.842598 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:33.844320 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:33.845005 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:33.845009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:33.845011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:33.845122 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:33.845570 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:33.845573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:33.845574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:33.845826 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:33.845912 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:33.846538 (+ 626us) server_negotiation.cc:299] Negotiation successful
0810 03:40:33.846706 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:40:33.847707 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.832826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36197 (local address 127.21.175.67:44151)
0810 03:40:33.833397 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:33.833442 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:33.834558 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:33.835194 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:33.835201 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:33.835673 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:33.836351 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.836367 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.837211 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.837215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.838298 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.838313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.839065 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.839071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.839311 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.840094 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:33.840117 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:33.842335 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:33.844461 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:33.844469 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:33.844487 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:33.844850 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:33.845245 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:33.845248 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:33.845251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:33.845415 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:33.845971 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:33.845976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:33.846226 (+ 250us) client_negotiation.cc:769] Sending connection context
0810 03:40:33.846405 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 03:40:33.846708 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":160,"threads_started":1}
I20260810 03:40:33.848796 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 30557
I20260810 03:40:33.848968 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:40:33.848999 30698 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36197
I20260810 03:40:33.849318 22205 external_mini_cluster.cc:1428] Setting key 2b58162c31faf31fb41e5a2c48471ed0
I20260810 03:40:33.851488 30698 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:33.852133 30698 heartbeater.cc:511] Master 127.21.175.126:36197 requested a full tablet report, sending...
I20260810 03:40:33.853091 30186 ts_manager.cc:194] Registered new tserver with Master: 00fe9b28af39403f8287d5ad827007e3 (127.21.175.67:39917)
I20260810 03:40:33.853828 30186 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:44151
I20260810 03:40:33.859463 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.851668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:59627 (local address 127.21.175.126:36197)
0810 03:40:33.851788 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:33.851791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:33.852035 (+ 244us) server_negotiation.cc:407] Connection header received
0810 03:40:33.852184 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:33.852188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:33.852234 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:33.852306 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:33.852833 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.853577 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.854577 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.854859 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.854944 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.856341 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:33.856345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:33.856348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:33.856376 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:33.858034 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:33.858455 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:33.858458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:33.858459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:33.858534 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:33.858805 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:33.858807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:33.858808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:33.858982 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:33.859072 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:33.859255 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 03:40:33.859350 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:40:33.859480 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.851668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36197 (local address 127.21.175.67:59627)
0810 03:40:33.851953 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:33.851965 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:33.852079 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:33.852322 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:33.852325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:33.852554 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:33.852692 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.852700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.853695 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.853697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.854450 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.854461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.854976 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.854980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.855066 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.855706 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:33.855720 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:33.856171 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:33.858157 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:33.858160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:33.858161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:33.858363 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:33.858648 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:33.858650 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:33.858652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:33.858727 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:33.859059 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:33.859061 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:33.859142 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 03:40:33.859221 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 03:40:33.859334 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":42,"thread_start_us":107,"threads_started":1}
I20260810 03:40:33.863336 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:40:33 dist-test-slave-ss4n krb5kdc[30137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:40:33 dist-test-slave-ss4n krb5kdc[30137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:33 dist-test-slave-ss4n krb5kdc[30137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 03:40:33.890789 30235 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:33.883682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52190 (local address 127.21.175.126:36197)
0810 03:40:33.883852 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:33.883857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:33.883886 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:40:33.883949 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:33.883952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:33.884016 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:33.884114 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:33.884949 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.885570 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.886498 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.886728 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.886841 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.890630 (+ 3789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52190: BlockingRecv error: recv got EOF from 127.0.0.1:52190 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 03:40:33 dist-test-slave-ss4n krb5kdc[30137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM
I20260810 03:40:33.901288 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.892358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.21.175.126:36197)
0810 03:40:33.892526 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:33.892531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:33.892705 (+ 174us) server_negotiation.cc:407] Connection header received
0810 03:40:33.892897 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:33.892900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:33.892944 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:33.893033 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:33.893792 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.894329 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.895171 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.895394 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.895504 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.898167 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:33.898171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:33.898173 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:33.898202 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:33.899513 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:33.899973 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:33.899977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:33.899979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:33.900069 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:33.900366 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:33.900370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:33.900372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:33.900573 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:33.900718 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:33.900970 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 03:40:33.901096 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:40:33.904619 30186 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:40:33.907820 30186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:40:33.925753 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.917803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:39665 (local address 127.0.0.1:59068)
0810 03:40:33.919256 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:33.919324 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:33.919469 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:33.920418 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:33.920426 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:33.920451 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:33.920610 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.920617 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.922342 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.922347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.923780 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.923791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.924990 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.924994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.925169 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.925305 (+ 136us) client_negotiation.cc:769] Sending connection context
0810 03:40:33.925421 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 03:40:33.925581 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1248,"mutex_wait_us":70,"thread_start_us":97,"threads_started":1}
I20260810 03:40:33.925799 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.918050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59068 (local address 127.21.175.66:39665)
0810 03:40:33.919852 (+ 1802us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:33.919864 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:33.919889 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:40:33.919971 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:33.919978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:33.920235 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:33.920777 (+ 542us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:33.920927 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.922201 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.923943 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.924840 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.925222 (+ 382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.925415 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:33.925471 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 03:40:33.925615 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1687,"thread_start_us":225,"threads_started":1}
I20260810 03:40:33.926762 30722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.918939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:41379 (local address 127.0.0.1:37424)
0810 03:40:33.920123 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:33.920137 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:33.920220 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:33.920800 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:33.920803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:33.920822 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:33.920971 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.920978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.922651 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.922655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.923814 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.923824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.925126 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.925130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.926432 (+ 1302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.926467 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 03:40:33.926554 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 03:40:33.926630 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1111,"mutex_wait_us":1135,"thread_start_us":86,"threads_started":1}
I20260810 03:40:33.927493 30727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.919367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.21.175.65:41379)
0810 03:40:33.920308 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:33.920315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:33.920339 (+ 24us) server_negotiation.cc:407] Connection header received
0810 03:40:33.920417 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:33.920422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:33.920627 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:33.921138 (+ 511us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:33.921285 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.922522 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.924169 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.924989 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.927021 (+ 2032us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.927189 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:33.927248 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 03:40:33.927359 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":96,"threads_started":1}
I20260810 03:40:33.927511 30726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.917903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:39917 (local address 127.0.0.1:50950)
0810 03:40:33.919939 (+ 2036us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:33.919962 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:33.920059 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:33.921161 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:33.921163 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:33.921181 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:33.921270 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.921275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.923617 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.923620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.925323 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:33.925356 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.927120 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.927123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:33.927208 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.927230 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 03:40:33.927306 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 03:40:33.927379 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1966,"mutex_wait_us":3,"thread_start_us":142,"threads_started":1}
I20260810 03:40:33.927600 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:33.918698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50950 (local address 127.21.175.67:39917)
0810 03:40:33.920494 (+ 1796us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:33.920507 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:33.920529 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:40:33.920615 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:33.920621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:33.920850 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:33.921007 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:33.922281 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.923426 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.925554 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:33.927007 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:33.927120 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:33.927294 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:33.927349 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 03:40:33.927453 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1698,"thread_start_us":178,"threads_started":1}
I20260810 03:40:33.929709 30478 tablet_service.cc:1467] Processing CreateTablet for tablet 119cc330abd44ffc9fc82b3002648db9 (DEFAULT_TABLE table=test-table [id=8bafbd1ad5bb4d1d850154084858e20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:33.930181 30330 tablet_service.cc:1467] Processing CreateTablet for tablet 119cc330abd44ffc9fc82b3002648db9 (DEFAULT_TABLE table=test-table [id=8bafbd1ad5bb4d1d850154084858e20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:33.930922 30478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 119cc330abd44ffc9fc82b3002648db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:33.930652 30628 tablet_service.cc:1467] Processing CreateTablet for tablet 119cc330abd44ffc9fc82b3002648db9 (DEFAULT_TABLE table=test-table [id=8bafbd1ad5bb4d1d850154084858e20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:33.931274 30330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 119cc330abd44ffc9fc82b3002648db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:33.931602 30628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 119cc330abd44ffc9fc82b3002648db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:33.937776 30728 tablet_bootstrap.cc:492] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3: Bootstrap starting.
I20260810 03:40:33.939464 30730 tablet_bootstrap.cc:492] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3: Bootstrap starting.
I20260810 03:40:33.940208 30729 tablet_bootstrap.cc:492] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c: Bootstrap starting.
I20260810 03:40:33.940382 30728 tablet_bootstrap.cc:654] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:33.941324 30728 log.cc:824] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:33.941707 30730 tablet_bootstrap.cc:654] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:33.942379 30729 tablet_bootstrap.cc:654] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:33.942624 30730 log.cc:824] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:33.943171 30729 log.cc:824] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:33.943725 30728 tablet_bootstrap.cc:492] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3: No bootstrap required, opened a new log
I20260810 03:40:33.943989 30728 ts_tablet_manager.cc:1397] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:40:33.944651 30730 tablet_bootstrap.cc:492] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3: No bootstrap required, opened a new log
I20260810 03:40:33.944880 30730 ts_tablet_manager.cc:1397] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:40:33.945161 30729 tablet_bootstrap.cc:492] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c: No bootstrap required, opened a new log
I20260810 03:40:33.945358 30729 ts_tablet_manager.cc:1397] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 03:40:33.947942 30730 raft_consensus.cc:348] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } }
I20260810 03:40:33.948218 30730 raft_consensus.cc:374] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:33.948274 30730 raft_consensus.cc:729] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe9b28af39403f8287d5ad827007e3, State: Initialized, Role: FOLLOWER
I20260810 03:40:33.948827 30730 consensus_queue.cc:260] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } }
I20260810 03:40:33.949407 30729 raft_consensus.cc:348] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } }
I20260810 03:40:33.949687 30698 heartbeater.cc:503] Master 127.21.175.126:36197 was elected leader, sending a full tablet report...
I20260810 03:40:33.949764 30729 raft_consensus.cc:374] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:33.949822 30729 raft_consensus.cc:729] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4962c93336104d7790c2f3e5d1ed5c8c, State: Initialized, Role: FOLLOWER
I20260810 03:40:33.949586 30728 raft_consensus.cc:348] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } }
I20260810 03:40:33.950066 30728 raft_consensus.cc:374] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:33.950163 30728 raft_consensus.cc:729] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122fedfa36e54e0eb3a9ed06808551f3, State: Initialized, Role: FOLLOWER
I20260810 03:40:33.950167 30730 ts_tablet_manager.cc:1428] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 03:40:33.950261 30729 consensus_queue.cc:260] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } }
I20260810 03:40:33.950767 30728 consensus_queue.cc:260] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } }
I20260810 03:40:33.951026 30547 heartbeater.cc:503] Master 127.21.175.126:36197 was elected leader, sending a full tablet report...
I20260810 03:40:33.951351 30729 ts_tablet_manager.cc:1428] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 03:40:33.951819 30728 ts_tablet_manager.cc:1428] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260810 03:40:33.952204 30399 heartbeater.cc:503] Master 127.21.175.126:36197 was elected leader, sending a full tablet report...
I20260810 03:40:34.066567 30734 raft_consensus.cc:482] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:34.066843 30734 raft_consensus.cc:504] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } }
I20260810 03:40:34.068105 30734 leader_election.cc:288] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4962c93336104d7790c2f3e5d1ed5c8c (127.21.175.66:39665), 122fedfa36e54e0eb3a9ed06808551f3 (127.21.175.65:41379)
I20260810 03:40:34.073113 30727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.068603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:48217 (local address 127.21.175.65:41379)
0810 03:40:34.068748 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:34.068751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:34.068767 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:40:34.068947 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:34.068951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:34.069009 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:34.069102 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:34.069394 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.070311 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.071829 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.072435 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.072577 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.072650 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:34.072943 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 03:40:34.073005 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:40:34.073168 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.068455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:41379 (local address 127.21.175.67:48217)
0810 03:40:34.068676 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:34.068698 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:34.068818 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:34.069116 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:34.069120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:34.069137 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:34.069262 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.069269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.070425 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.070430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.071691 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.071702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.072646 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.072650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.072787 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.072818 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 03:40:34.072932 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 03:40:34.073011 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260810 03:40:34.073982 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.068596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:59593 (local address 127.21.175.66:39665)
0810 03:40:34.068727 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:34.068734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:34.069223 (+ 489us) server_negotiation.cc:407] Connection header received
0810 03:40:34.069325 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:34.069330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:34.069387 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:34.069456 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:34.070357 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.071189 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.072666 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.073330 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.073472 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.073579 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:34.073785 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 03:40:34.073876 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:40:34.073977 30737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.068455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:39665 (local address 127.21.175.67:59593)
0810 03:40:34.069140 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:34.069152 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:34.069245 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:34.069523 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:34.069526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:34.069546 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:34.070142 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.070152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.071373 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.071376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.072486 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.072499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.073520 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.073524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.073652 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.073672 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 03:40:34.073764 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 03:40:34.073832 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":323,"thread_start_us":148,"threads_started":1}
I20260810 03:40:34.073949 30350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119cc330abd44ffc9fc82b3002648db9" candidate_uuid: "00fe9b28af39403f8287d5ad827007e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122fedfa36e54e0eb3a9ed06808551f3" is_pre_election: true
I20260810 03:40:34.074368 30350 raft_consensus.cc:2454] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe9b28af39403f8287d5ad827007e3 in term 0.
I20260810 03:40:34.074700 30499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119cc330abd44ffc9fc82b3002648db9" candidate_uuid: "00fe9b28af39403f8287d5ad827007e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" is_pre_election: true
I20260810 03:40:34.075002 30499 raft_consensus.cc:2454] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe9b28af39403f8287d5ad827007e3 in term 0.
I20260810 03:40:34.074985 30577 leader_election.cc:302] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe9b28af39403f8287d5ad827007e3, 122fedfa36e54e0eb3a9ed06808551f3; no voters:
I20260810 03:40:34.075314 30734 raft_consensus.cc:2781] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:40:34.075412 30734 raft_consensus.cc:482] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:40:34.075487 30734 raft_consensus.cc:3037] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:34.076759 30734 raft_consensus.cc:504] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } }
I20260810 03:40:34.077219 30734 leader_election.cc:288] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [CANDIDATE]: Term 1 election: Requested vote from peers 4962c93336104d7790c2f3e5d1ed5c8c (127.21.175.66:39665), 122fedfa36e54e0eb3a9ed06808551f3 (127.21.175.65:41379)
I20260810 03:40:34.077615 30350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119cc330abd44ffc9fc82b3002648db9" candidate_uuid: "00fe9b28af39403f8287d5ad827007e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122fedfa36e54e0eb3a9ed06808551f3"
I20260810 03:40:34.077596 30499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119cc330abd44ffc9fc82b3002648db9" candidate_uuid: "00fe9b28af39403f8287d5ad827007e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4962c93336104d7790c2f3e5d1ed5c8c"
I20260810 03:40:34.077759 30499 raft_consensus.cc:3037] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:34.077795 30350 raft_consensus.cc:3037] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:34.079218 30499 raft_consensus.cc:2454] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe9b28af39403f8287d5ad827007e3 in term 1.
I20260810 03:40:34.079218 30350 raft_consensus.cc:2454] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe9b28af39403f8287d5ad827007e3 in term 1.
I20260810 03:40:34.079571 30577 leader_election.cc:302] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe9b28af39403f8287d5ad827007e3, 122fedfa36e54e0eb3a9ed06808551f3; no voters:
I20260810 03:40:34.079834 30734 raft_consensus.cc:2781] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:34.080154 30734 raft_consensus.cc:686] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [term 1 LEADER]: Becoming Leader. State: Replica: 00fe9b28af39403f8287d5ad827007e3, State: Running, Role: LEADER
I20260810 03:40:34.080510 30734 consensus_queue.cc:237] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } }
I20260810 03:40:34.083884 30186 catalog_manager.cc:5495] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe9b28af39403f8287d5ad827007e3 (127.21.175.67). New cstate: current_term: 1 leader_uuid: "00fe9b28af39403f8287d5ad827007e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } health_report { overall_health: UNKNOWN } } }
W20260810 03:40:34.095489 30699 tablet.cc:2369] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:40:34.097755 30548 tablet.cc:2369] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:40:34.100045 30400 tablet.cc:2369] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:40:34.117187 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.113636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.21.175.67:39917)
0810 03:40:34.113783 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:34.113788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:34.113805 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:40:34.113902 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:34.113904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:34.113959 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:34.114094 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:40:34.114340 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.114845 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.115700 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.115929 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.116040 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.116328 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:40:34.116779 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:40:34.116908 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:34.116978 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 03:40:34.117038 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:40:34.123543 30499 raft_consensus.cc:1261] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Refusing update from remote peer 00fe9b28af39403f8287d5ad827007e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:34.123594 30350 raft_consensus.cc:1261] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 FOLLOWER]: Refusing update from remote peer 00fe9b28af39403f8287d5ad827007e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:34.124311 30738 consensus_queue.cc:1035] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:34.124506 30734 consensus_queue.cc:1035] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:34.132304 30744 mvcc.cc:205] Tried to move back new op lower bound from 7316820926962040832 to 7316820926796587008. Current Snapshot: MvccSnapshot[applied={T|T < 7316820926962040832}]
I20260810 03:40:34.136620 30743 mvcc.cc:205] Tried to move back new op lower bound from 7316820926962040832 to 7316820926796587008. Current Snapshot: MvccSnapshot[applied={T|T < 7316820926962040832 or (T in {7316820926962040832})}]
I20260810 03:40:34.141928 30186 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:34.142094 30186 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:34.144647 30186 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c: Sending DeleteTablet for 3 replicas of tablet 119cc330abd44ffc9fc82b3002648db9
I20260810 03:40:34.145588 30628 tablet_service.cc:1514] Processing DeleteTablet for tablet 119cc330abd44ffc9fc82b3002648db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:34 UTC) from {username='oryx', principal='oryx/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:50950
I20260810 03:40:34.145537 30478 tablet_service.cc:1514] Processing DeleteTablet for tablet 119cc330abd44ffc9fc82b3002648db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:34 UTC) from {username='oryx', principal='oryx/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:59068
I20260810 03:40:34.146108 30752 tablet_replica.cc:330] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c: stopping tablet replica
I20260810 03:40:34.146117 30330 tablet_service.cc:1514] Processing DeleteTablet for tablet 119cc330abd44ffc9fc82b3002648db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:34 UTC) from {username='oryx', principal='oryx/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:37424
I20260810 03:40:34.146476 30752 raft_consensus.cc:2229] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:34.146600 30754 tablet_replica.cc:330] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3: stopping tablet replica
I20260810 03:40:34.146811 30752 raft_consensus.cc:2258] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:34.146899 30754 raft_consensus.cc:2229] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:34.147195 30753 tablet_replica.cc:330] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3: stopping tablet replica
I20260810 03:40:34.147205 30754 raft_consensus.cc:2258] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:34.147478 30753 raft_consensus.cc:2229] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:40:34.147442 30186 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:40:34.147826 30186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:40:34.147830 30753 raft_consensus.cc:2258] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:34.150022 30754 ts_tablet_manager.cc:1905] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:34.151010 30753 ts_tablet_manager.cc:1905] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:34.152555 30752 ts_tablet_manager.cc:1905] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:34.153215 30754 ts_tablet_manager.cc:1918] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:40:34.153326 30754 log.cc:1196] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/wal/wals/119cc330abd44ffc9fc82b3002648db9
I20260810 03:40:34.153697 30754 ts_tablet_manager.cc:1939] T 119cc330abd44ffc9fc82b3002648db9 P 122fedfa36e54e0eb3a9ed06808551f3: Deleting consensus metadata
I20260810 03:40:34.153941 30330 tablet_service.cc:1467] Processing CreateTablet for tablet 99ea9d296c2c47c8b5f5af7deceb9c83 (DEFAULT_TABLE table=test-table [id=fbb48fa870414ae0ab0fea6f4c40af67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:34.154182 30330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ea9d296c2c47c8b5f5af7deceb9c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:34.154999 30169 catalog_manager.cc:4841] TS 122fedfa36e54e0eb3a9ed06808551f3 (127.21.175.65:41379): tablet 119cc330abd44ffc9fc82b3002648db9 (table test-table [id=8bafbd1ad5bb4d1d850154084858e20a]) successfully deleted
I20260810 03:40:34.155288 30752 ts_tablet_manager.cc:1918] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:40:34.155381 30752 log.cc:1196] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/119cc330abd44ffc9fc82b3002648db9
I20260810 03:40:34.155674 30752 ts_tablet_manager.cc:1939] T 119cc330abd44ffc9fc82b3002648db9 P 4962c93336104d7790c2f3e5d1ed5c8c: Deleting consensus metadata
I20260810 03:40:34.156361 30728 tablet_bootstrap.cc:492] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3: Bootstrap starting.
I20260810 03:40:34.157225 30728 tablet_bootstrap.cc:654] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:34.158363 30628 tablet_service.cc:1467] Processing CreateTablet for tablet 99ea9d296c2c47c8b5f5af7deceb9c83 (DEFAULT_TABLE table=test-table [id=fbb48fa870414ae0ab0fea6f4c40af67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:34.158526 30753 ts_tablet_manager.cc:1918] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:40:34.158620 30753 log.cc:1196] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/119cc330abd44ffc9fc82b3002648db9
I20260810 03:40:34.158651 30628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ea9d296c2c47c8b5f5af7deceb9c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:34.158931 30753 ts_tablet_manager.cc:1939] T 119cc330abd44ffc9fc82b3002648db9 P 00fe9b28af39403f8287d5ad827007e3: Deleting consensus metadata
I20260810 03:40:34.159046 30478 tablet_service.cc:1467] Processing CreateTablet for tablet 99ea9d296c2c47c8b5f5af7deceb9c83 (DEFAULT_TABLE table=test-table [id=fbb48fa870414ae0ab0fea6f4c40af67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:34.159292 30478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ea9d296c2c47c8b5f5af7deceb9c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:34.160920 30730 tablet_bootstrap.cc:492] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3: Bootstrap starting.
I20260810 03:40:34.161684 30729 tablet_bootstrap.cc:492] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c: Bootstrap starting.
I20260810 03:40:34.161978 30730 tablet_bootstrap.cc:654] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:34.162048 30167 catalog_manager.cc:4841] TS 4962c93336104d7790c2f3e5d1ed5c8c (127.21.175.66:39665): tablet 119cc330abd44ffc9fc82b3002648db9 (table test-table [id=8bafbd1ad5bb4d1d850154084858e20a]) successfully deleted
I20260810 03:40:34.162659 30729 tablet_bootstrap.cc:654] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:34.162928 30168 catalog_manager.cc:4841] TS 00fe9b28af39403f8287d5ad827007e3 (127.21.175.67:39917): tablet 119cc330abd44ffc9fc82b3002648db9 (table test-table [id=8bafbd1ad5bb4d1d850154084858e20a]) successfully deleted
I20260810 03:40:34.163401 30730 tablet_bootstrap.cc:492] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3: No bootstrap required, opened a new log
I20260810 03:40:34.163489 30730 ts_tablet_manager.cc:1397] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:34.163991 30730 raft_consensus.cc:348] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.164132 30730 raft_consensus.cc:374] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:34.164176 30730 raft_consensus.cc:729] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe9b28af39403f8287d5ad827007e3, State: Initialized, Role: FOLLOWER
I20260810 03:40:34.164235 30728 tablet_bootstrap.cc:492] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3: No bootstrap required, opened a new log
I20260810 03:40:34.164330 30728 ts_tablet_manager.cc:1397] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260810 03:40:34.164300 30730 consensus_queue.cc:260] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.164646 30730 ts_tablet_manager.cc:1428] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:40:34.165241 30728 raft_consensus.cc:348] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.165390 30728 raft_consensus.cc:374] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:34.165432 30728 raft_consensus.cc:729] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122fedfa36e54e0eb3a9ed06808551f3, State: Initialized, Role: FOLLOWER
I20260810 03:40:34.165587 30728 consensus_queue.cc:260] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.165907 30728 ts_tablet_manager.cc:1428] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:40:34.168102 30729 tablet_bootstrap.cc:492] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c: No bootstrap required, opened a new log
I20260810 03:40:34.168226 30729 ts_tablet_manager.cc:1397] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260810 03:40:34.168731 30729 raft_consensus.cc:348] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.168864 30729 raft_consensus.cc:374] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:34.168916 30729 raft_consensus.cc:729] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4962c93336104d7790c2f3e5d1ed5c8c, State: Initialized, Role: FOLLOWER
I20260810 03:40:34.169629 30729 consensus_queue.cc:260] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.169938 30729 ts_tablet_manager.cc:1428] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 03:40:34.172128 30748 raft_consensus.cc:482] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:34.172243 30748 raft_consensus.cc:504] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.172629 30748 leader_election.cc:288] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 122fedfa36e54e0eb3a9ed06808551f3 (127.21.175.65:41379), 4962c93336104d7790c2f3e5d1ed5c8c (127.21.175.66:39665)
I20260810 03:40:34.173012 30499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ea9d296c2c47c8b5f5af7deceb9c83" candidate_uuid: "00fe9b28af39403f8287d5ad827007e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" is_pre_election: true
I20260810 03:40:34.173050 30350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ea9d296c2c47c8b5f5af7deceb9c83" candidate_uuid: "00fe9b28af39403f8287d5ad827007e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122fedfa36e54e0eb3a9ed06808551f3" is_pre_election: true
I20260810 03:40:34.173139 30499 raft_consensus.cc:2454] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe9b28af39403f8287d5ad827007e3 in term 0.
I20260810 03:40:34.173215 30350 raft_consensus.cc:2454] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe9b28af39403f8287d5ad827007e3 in term 0.
I20260810 03:40:34.173441 30574 leader_election.cc:302] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe9b28af39403f8287d5ad827007e3, 4962c93336104d7790c2f3e5d1ed5c8c; no voters:
I20260810 03:40:34.173662 30748 raft_consensus.cc:2781] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:40:34.173717 30748 raft_consensus.cc:482] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:40:34.173753 30748 raft_consensus.cc:3037] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:34.174973 30748 raft_consensus.cc:504] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.175315 30748 leader_election.cc:288] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [CANDIDATE]: Term 1 election: Requested vote from peers 122fedfa36e54e0eb3a9ed06808551f3 (127.21.175.65:41379), 4962c93336104d7790c2f3e5d1ed5c8c (127.21.175.66:39665)
I20260810 03:40:34.175658 30350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ea9d296c2c47c8b5f5af7deceb9c83" candidate_uuid: "00fe9b28af39403f8287d5ad827007e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122fedfa36e54e0eb3a9ed06808551f3"
I20260810 03:40:34.175755 30499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ea9d296c2c47c8b5f5af7deceb9c83" candidate_uuid: "00fe9b28af39403f8287d5ad827007e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4962c93336104d7790c2f3e5d1ed5c8c"
I20260810 03:40:34.175786 30350 raft_consensus.cc:3037] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:34.175856 30499 raft_consensus.cc:3037] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:34.176963 30350 raft_consensus.cc:2454] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe9b28af39403f8287d5ad827007e3 in term 1.
I20260810 03:40:34.176963 30499 raft_consensus.cc:2454] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe9b28af39403f8287d5ad827007e3 in term 1.
I20260810 03:40:34.177289 30574 leader_election.cc:302] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe9b28af39403f8287d5ad827007e3, 4962c93336104d7790c2f3e5d1ed5c8c; no voters:
I20260810 03:40:34.177613 30748 raft_consensus.cc:2781] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:34.177719 30748 raft_consensus.cc:686] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [term 1 LEADER]: Becoming Leader. State: Replica: 00fe9b28af39403f8287d5ad827007e3, State: Running, Role: LEADER
I20260810 03:40:34.177853 30748 consensus_queue.cc:237] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.179461 30185 catalog_manager.cc:5495] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe9b28af39403f8287d5ad827007e3 (127.21.175.67). New cstate: current_term: 1 leader_uuid: "00fe9b28af39403f8287d5ad827007e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } health_report { overall_health: HEALTHY } } }
Aug 10 03:40:34 dist-test-slave-ss4n krb5kdc[30137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333233, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.126@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM
I20260810 03:40:34.220710 30765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.211415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36197 (local address 127.0.0.1:52208)
0810 03:40:34.211759 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:34.211772 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:34.211910 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:34.212190 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:34.212193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:34.212633 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:34.212734 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.212741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.213505 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.213511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.214071 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.214078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.214523 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.214526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.214603 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.215331 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:34.215343 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:34.217059 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:34.218903 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:34.218914 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:34.218922 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:34.219187 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:34.219478 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:34.219481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:34.219483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:34.219569 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:34.219989 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:34.219998 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:34.220185 (+ 187us) client_negotiation.cc:769] Sending connection context
0810 03:40:34.220323 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 03:40:34.220500 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":130,"threads_started":1}
I20260810 03:40:34.221055 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.211484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.21.175.126:36197)
0810 03:40:34.211676 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:34.211681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:34.211848 (+ 167us) server_negotiation.cc:407] Connection header received
0810 03:40:34.212030 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:34.212037 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:34.212073 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:34.212157 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:34.212857 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.213378 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.214198 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.214410 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.214519 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.217231 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:34.217235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:34.217237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:34.217258 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:34.218773 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:34.219293 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:34.219298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:34.219300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:34.219378 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:34.219684 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:34.219689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:34.219691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:34.219881 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:34.219989 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:34.220395 (+ 406us) server_negotiation.cc:299] Negotiation successful
0810 03:40:34.220612 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":47}
I20260810 03:40:34.226655 30186 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:52208:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 03:40:34.227479 30186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:40:34.233857 30330 tablet_service.cc:1467] Processing CreateTablet for tablet dfc1cffcef15485c8f3968959cfec6e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dea6a48ac4cf4855ac20caccdd841ceb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:40:34.234122 30628 tablet_service.cc:1467] Processing CreateTablet for tablet dfc1cffcef15485c8f3968959cfec6e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dea6a48ac4cf4855ac20caccdd841ceb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:40:34.234131 30478 tablet_service.cc:1467] Processing CreateTablet for tablet dfc1cffcef15485c8f3968959cfec6e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dea6a48ac4cf4855ac20caccdd841ceb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:40:34.234311 30330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc1cffcef15485c8f3968959cfec6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:34.234498 30478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc1cffcef15485c8f3968959cfec6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:34.234498 30628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc1cffcef15485c8f3968959cfec6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:34.237417 30728 tablet_bootstrap.cc:492] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3: Bootstrap starting.
I20260810 03:40:34.237798 30730 tablet_bootstrap.cc:492] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3: Bootstrap starting.
I20260810 03:40:34.238473 30728 tablet_bootstrap.cc:654] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:34.238950 30730 tablet_bootstrap.cc:654] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:34.239166 30729 tablet_bootstrap.cc:492] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c: Bootstrap starting.
I20260810 03:40:34.240108 30729 tablet_bootstrap.cc:654] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:34.240500 30730 tablet_bootstrap.cc:492] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3: No bootstrap required, opened a new log
I20260810 03:40:34.240594 30730 ts_tablet_manager.cc:1397] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:34.240996 30728 tablet_bootstrap.cc:492] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3: No bootstrap required, opened a new log
I20260810 03:40:34.241072 30728 ts_tablet_manager.cc:1397] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 03:40:34.241083 30730 raft_consensus.cc:348] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.241215 30730 raft_consensus.cc:374] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:34.241255 30730 raft_consensus.cc:729] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe9b28af39403f8287d5ad827007e3, State: Initialized, Role: FOLLOWER
I20260810 03:40:34.241391 30730 consensus_queue.cc:260] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.241473 30728 raft_consensus.cc:348] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.241616 30728 raft_consensus.cc:374] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:34.241645 30728 raft_consensus.cc:729] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122fedfa36e54e0eb3a9ed06808551f3, State: Initialized, Role: FOLLOWER
I20260810 03:40:34.241669 30748 tablet_replica.cc:439] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } }
I20260810 03:40:34.241763 30728 consensus_queue.cc:260] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.241814 30748 tablet_replica.cc:442] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:40:34.242033 30736 tablet_replica.cc:439] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } }
I20260810 03:40:34.242228 30736 tablet_replica.cc:442] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:40:34.242031 30728 ts_tablet_manager.cc:1428] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:40:34.243000 30729 tablet_bootstrap.cc:492] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c: No bootstrap required, opened a new log
I20260810 03:40:34.243083 30729 ts_tablet_manager.cc:1397] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 03:40:34.243112 30730 ts_tablet_manager.cc:1428] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 03:40:34.243515 30729 raft_consensus.cc:348] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.243621 30729 raft_consensus.cc:374] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:34.243650 30729 raft_consensus.cc:729] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4962c93336104d7790c2f3e5d1ed5c8c, State: Initialized, Role: FOLLOWER
I20260810 03:40:34.243785 30729 consensus_queue.cc:260] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.244040 30729 ts_tablet_manager.cc:1428] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 03:40:34.244313 30735 tablet_replica.cc:439] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } }
I20260810 03:40:34.244467 30735 tablet_replica.cc:442] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:40:34.448792 30735 raft_consensus.cc:482] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:34.448969 30735 raft_consensus.cc:504] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.450238 30735 leader_election.cc:288] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 122fedfa36e54e0eb3a9ed06808551f3 (127.21.175.65:41379), 00fe9b28af39403f8287d5ad827007e3 (127.21.175.67:39917)
I20260810 03:40:34.454438 30771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.450478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:41379 (local address 127.21.175.66:35319)
0810 03:40:34.450930 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:34.450943 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:34.451049 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:34.451313 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:34.451316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:34.451335 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:34.451467 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.451474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.452308 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.452311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.453290 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.453299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.454000 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.454007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.454096 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.454119 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 03:40:34.454196 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 03:40:34.454252 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":73,"thread_start_us":157,"threads_started":1}
I20260810 03:40:34.454458 30727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.450615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:35319 (local address 127.21.175.65:41379)
0810 03:40:34.450825 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:34.450830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:34.451030 (+ 200us) server_negotiation.cc:407] Connection header received
0810 03:40:34.451142 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:34.451148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:34.451202 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:34.451296 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:34.451591 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.452192 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.453432 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.453876 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.453976 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.454058 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:34.454233 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 03:40:34.454329 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260810 03:40:34.454562 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.450627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:60621 (local address 127.21.175.67:39917)
0810 03:40:34.450803 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:34.450810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:34.451183 (+ 373us) server_negotiation.cc:407] Connection header received
0810 03:40:34.451316 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:34.451319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:34.451370 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:34.451453 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:34.451796 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.452325 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.453520 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.453951 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.454069 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.454131 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:34.454354 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 03:40:34.454429 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:40:34.454591 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.450544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:39917 (local address 127.21.175.66:60621)
0810 03:40:34.451064 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:34.451078 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:34.451188 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:34.451540 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:34.451545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:34.451569 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:34.451683 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.451690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.452473 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.452476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.453318 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.453329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.454095 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.454099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.454218 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.454245 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 03:40:34.454323 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 03:40:34.454385 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":113,"threads_started":1}
I20260810 03:40:34.455137 30350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc1cffcef15485c8f3968959cfec6e8" candidate_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122fedfa36e54e0eb3a9ed06808551f3" is_pre_election: true
I20260810 03:40:34.455302 30350 raft_consensus.cc:2454] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4962c93336104d7790c2f3e5d1ed5c8c in term 0.
I20260810 03:40:34.455673 30650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc1cffcef15485c8f3968959cfec6e8" candidate_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe9b28af39403f8287d5ad827007e3" is_pre_election: true
I20260810 03:40:34.455740 30427 leader_election.cc:302] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 122fedfa36e54e0eb3a9ed06808551f3, 4962c93336104d7790c2f3e5d1ed5c8c; no voters:
I20260810 03:40:34.455932 30650 raft_consensus.cc:2454] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4962c93336104d7790c2f3e5d1ed5c8c in term 0.
I20260810 03:40:34.456070 30735 raft_consensus.cc:2781] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:40:34.456167 30735 raft_consensus.cc:482] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:40:34.456223 30735 raft_consensus.cc:3037] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:34.457355 30735 raft_consensus.cc:504] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.457777 30735 leader_election.cc:288] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [CANDIDATE]: Term 1 election: Requested vote from peers 122fedfa36e54e0eb3a9ed06808551f3 (127.21.175.65:41379), 00fe9b28af39403f8287d5ad827007e3 (127.21.175.67:39917)
I20260810 03:40:34.458146 30350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc1cffcef15485c8f3968959cfec6e8" candidate_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122fedfa36e54e0eb3a9ed06808551f3"
I20260810 03:40:34.458177 30650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc1cffcef15485c8f3968959cfec6e8" candidate_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe9b28af39403f8287d5ad827007e3"
I20260810 03:40:34.458319 30350 raft_consensus.cc:3037] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:34.458324 30650 raft_consensus.cc:3037] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:34.459259 30350 raft_consensus.cc:2454] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4962c93336104d7790c2f3e5d1ed5c8c in term 1.
I20260810 03:40:34.459406 30650 raft_consensus.cc:2454] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4962c93336104d7790c2f3e5d1ed5c8c in term 1.
I20260810 03:40:34.459620 30427 leader_election.cc:302] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 122fedfa36e54e0eb3a9ed06808551f3, 4962c93336104d7790c2f3e5d1ed5c8c; no voters:
I20260810 03:40:34.459815 30735 raft_consensus.cc:2781] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:34.460021 30735 raft_consensus.cc:686] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 LEADER]: Becoming Leader. State: Replica: 4962c93336104d7790c2f3e5d1ed5c8c, State: Running, Role: LEADER
I20260810 03:40:34.460295 30735 consensus_queue.cc:237] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.461663 30772 tablet_replica.cc:439] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c: TxnStatusTablet state changed. Reason: New leader 4962c93336104d7790c2f3e5d1ed5c8c. Latest consensus state: current_term: 1 leader_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } }
I20260810 03:40:34.461794 30772 tablet_replica.cc:442] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:40:34.462074 30774 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 03:40:34.462256 30774 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 03:40:34.463126 30185 catalog_manager.cc:5495] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4962c93336104d7790c2f3e5d1ed5c8c (127.21.175.66). New cstate: current_term: 1 leader_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } health_report { overall_health: UNKNOWN } } }
I20260810 03:40:34.486253 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.482280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59072 (local address 127.21.175.66:39665)
0810 03:40:34.482442 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:34.482448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:34.482465 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:40:34.482531 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:34.482533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:34.482584 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:34.482693 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:40:34.482959 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.483470 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.484446 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.484716 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.484828 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.485178 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:40:34.485774 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:40:34.485936 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:34.486010 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 03:40:34.486069 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:40:34.486222 30765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.482003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:39665 (local address 127.0.0.1:59072)
0810 03:40:34.482188 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:34.482203 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:34.482300 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:34.482708 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:34.482712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:34.482720 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:40:34.482835 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.482848 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.483583 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.483587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.484300 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.484309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.484831 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.484835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.484934 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.484983 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:40:34.485898 (+ 915us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:40:34.485917 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 03:40:34.485998 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 03:40:34.486101 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 03:40:34.489607 30350 raft_consensus.cc:1261] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 FOLLOWER]: Refusing update from remote peer 4962c93336104d7790c2f3e5d1ed5c8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:34.489741 30650 raft_consensus.cc:1261] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3 [term 1 FOLLOWER]: Refusing update from remote peer 4962c93336104d7790c2f3e5d1ed5c8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:34.490135 30772 consensus_queue.cc:1035] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:34.490339 30735 consensus_queue.cc:1035] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:34.492031 30736 tablet_replica.cc:439] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3: TxnStatusTablet state changed. Reason: New leader 4962c93336104d7790c2f3e5d1ed5c8c. Latest consensus state: current_term: 1 leader_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } }
I20260810 03:40:34.492166 30736 tablet_replica.cc:442] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:40:34.494158 30735 tablet_replica.cc:439] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } }
I20260810 03:40:34.494305 30735 tablet_replica.cc:442] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:40:34.494282 30748 tablet_replica.cc:439] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3: TxnStatusTablet state changed. Reason: New leader 4962c93336104d7790c2f3e5d1ed5c8c. Latest consensus state: current_term: 1 leader_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } }
I20260810 03:40:34.494416 30748 tablet_replica.cc:442] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:40:34.495411 30736 tablet_replica.cc:439] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } }
I20260810 03:40:34.495605 30736 tablet_replica.cc:442] T dfc1cffcef15485c8f3968959cfec6e8 P 122fedfa36e54e0eb3a9ed06808551f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:40:34.499965 30772 tablet_replica.cc:439] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } }
I20260810 03:40:34.500128 30772 tablet_replica.cc:442] T dfc1cffcef15485c8f3968959cfec6e8 P 4962c93336104d7790c2f3e5d1ed5c8c: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:40:34.500828 30742 mvcc.cc:205] Tried to move back new op lower bound from 7316820928463265792 to 7316820928352014336. Current Snapshot: MvccSnapshot[applied={T|T < 7316820928463265792}]
I20260810 03:40:34.501016 30748 tablet_replica.cc:439] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } } }
I20260810 03:40:34.501134 30748 tablet_replica.cc:442] T dfc1cffcef15485c8f3968959cfec6e8 P 00fe9b28af39403f8287d5ad827007e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:40:34.511922 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.508124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:39665 (local address 127.21.175.67:42139)
0810 03:40:34.508424 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:34.508445 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:34.508541 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:34.508798 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:34.508800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:34.508806 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:40:34.508900 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.508904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.509958 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.509962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.510622 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.510632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.511222 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.511224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.511293 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.511345 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:40:34.511663 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:40:34.511668 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 03:40:34.511721 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 03:40:34.511796 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":109,"threads_started":1}
I20260810 03:40:34.511931 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.508244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:42139 (local address 127.21.175.66:39665)
0810 03:40:34.508379 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:34.508383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:34.508519 (+ 136us) server_negotiation.cc:407] Connection header received
0810 03:40:34.508638 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:34.508642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:34.508703 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:34.508797 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:40:34.508995 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.509834 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.510757 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.511104 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.511211 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.511486 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:40:34.511576 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:40:34.511648 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:34.511731 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 03:40:34.511788 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:40:34.519840 30499 raft_consensus.cc:1261] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Refusing update from remote peer 00fe9b28af39403f8287d5ad827007e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:34.520067 30350 raft_consensus.cc:1261] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 FOLLOWER]: Refusing update from remote peer 00fe9b28af39403f8287d5ad827007e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:34.520375 30738 consensus_queue.cc:1035] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:34.520815 30738 consensus_queue.cc:1035] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:34.545351 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.541038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:39917 (local address 127.21.175.66:50029)
0810 03:40:34.541435 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:34.541451 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:34.541595 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:34.541901 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:34.541904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:34.541933 (+ 29us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:40:34.542063 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.542074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.543016 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.543022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.543932 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.543942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.544538 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.544542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.544630 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.544683 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:40:34.545029 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:40:34.545036 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:40:34.545134 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 03:40:34.545206 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":121,"threads_started":1}
I20260810 03:40:34.545368 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.541447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:50029 (local address 127.21.175.67:39917)
0810 03:40:34.541674 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:34.541681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:34.541695 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:40:34.541741 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:34.541744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:34.541792 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:34.541876 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:40:34.542191 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.542904 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.544062 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.544410 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.544529 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.544843 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:40:34.544929 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:40:34.545002 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:34.545172 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 03:40:34.545235 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 03:40:34.571611 30186 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:34.571800 30186 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:34.574082 30186 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c: Sending DeleteTablet for 3 replicas of tablet 99ea9d296c2c47c8b5f5af7deceb9c83
I20260810 03:40:34.574826 30330 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ea9d296c2c47c8b5f5af7deceb9c83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:34 UTC) from {username='oryx', principal='oryx/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:37424
I20260810 03:40:34.575242 30478 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ea9d296c2c47c8b5f5af7deceb9c83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:34 UTC) from {username='oryx', principal='oryx/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:59068
I20260810 03:40:34.575343 30754 tablet_replica.cc:330] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3: stopping tablet replica
I20260810 03:40:34.575438 30752 tablet_replica.cc:330] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c: stopping tablet replica
I20260810 03:40:34.575513 30754 raft_consensus.cc:2229] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:34.575557 30752 raft_consensus.cc:2229] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:34.575670 30754 raft_consensus.cc:2258] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:34.575685 30752 raft_consensus.cc:2258] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:34.576051 30628 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ea9d296c2c47c8b5f5af7deceb9c83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:34 UTC) from {username='oryx', principal='oryx/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:50950
I20260810 03:40:34.576251 30753 tablet_replica.cc:330] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3: stopping tablet replica
I20260810 03:40:34.576367 30753 raft_consensus.cc:2229] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:40:34.576560 30752 ts_tablet_manager.cc:1905] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:34.576591 30753 raft_consensus.cc:2258] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:34.577374 30753 ts_tablet_manager.cc:1905] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:34.577503 30754 ts_tablet_manager.cc:1905] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:34.579676 30752 ts_tablet_manager.cc:1918] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:40:34.579764 30752 log.cc:1196] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/99ea9d296c2c47c8b5f5af7deceb9c83
I20260810 03:40:34.580018 30752 ts_tablet_manager.cc:1939] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 4962c93336104d7790c2f3e5d1ed5c8c: Deleting consensus metadata
I20260810 03:40:34.580181 30754 ts_tablet_manager.cc:1918] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:40:34.580181 30753 ts_tablet_manager.cc:1918] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:40:34.580255 30753 log.cc:1196] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/99ea9d296c2c47c8b5f5af7deceb9c83
I20260810 03:40:34.580255 30754 log.cc:1196] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/wal/wals/99ea9d296c2c47c8b5f5af7deceb9c83
I20260810 03:40:34.580519 30754 ts_tablet_manager.cc:1939] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 122fedfa36e54e0eb3a9ed06808551f3: Deleting consensus metadata
I20260810 03:40:34.580519 30753 ts_tablet_manager.cc:1939] T 99ea9d296c2c47c8b5f5af7deceb9c83 P 00fe9b28af39403f8287d5ad827007e3: Deleting consensus metadata
I20260810 03:40:34.580904 30167 catalog_manager.cc:4841] TS 4962c93336104d7790c2f3e5d1ed5c8c (127.21.175.66:39665): tablet 99ea9d296c2c47c8b5f5af7deceb9c83 (table test-table [id=fbb48fa870414ae0ab0fea6f4c40af67]) successfully deleted
I20260810 03:40:34.581213 30169 catalog_manager.cc:4841] TS 122fedfa36e54e0eb3a9ed06808551f3 (127.21.175.65:41379): tablet 99ea9d296c2c47c8b5f5af7deceb9c83 (table test-table [id=fbb48fa870414ae0ab0fea6f4c40af67]) successfully deleted
I20260810 03:40:34.583863 30168 catalog_manager.cc:4841] TS 00fe9b28af39403f8287d5ad827007e3 (127.21.175.67:39917): tablet 99ea9d296c2c47c8b5f5af7deceb9c83 (table test-table [id=fbb48fa870414ae0ab0fea6f4c40af67]) successfully deleted
Aug 10 03:40:34 dist-test-slave-ss4n krb5kdc[30137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:40:34 dist-test-slave-ss4n krb5kdc[30137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:34 dist-test-slave-ss4n krb5kdc[30137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 03:40:34.612815 30235 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:34.604418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52220 (local address 127.21.175.126:36197)
0810 03:40:34.604549 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:34.604554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:34.604861 (+ 307us) server_negotiation.cc:407] Connection header received
0810 03:40:34.605006 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:34.605009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:34.605062 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:34.605167 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:34.606203 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.607057 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.608321 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.608679 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.608784 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.612688 (+ 3904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52220: BlockingRecv error: recv got EOF from 127.0.0.1:52220 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 03:40:34 dist-test-slave-ss4n krb5kdc[30137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM
I20260810 03:40:34.625409 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.614995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.21.175.126:36197)
0810 03:40:34.615127 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:34.615131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:34.615535 (+ 404us) server_negotiation.cc:407] Connection header received
0810 03:40:34.615655 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:34.615659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:34.615715 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:34.615804 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:34.616599 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.617379 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.618502 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.618846 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.618987 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.621738 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:34.621742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:34.621745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:34.621781 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:34.623620 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:34.624057 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:34.624061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:34.624063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:34.624185 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:34.624481 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:34.624484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:34.624486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:34.624722 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:34.624865 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:34.625116 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 03:40:34.625238 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 03:40:34.628414 30186 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:40:34.628937 30186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:40:34.634377 30478 tablet_service.cc:1467] Processing CreateTablet for tablet 8b637b5460b7481884ab4945ee8bd23e (DEFAULT_TABLE table=test-table [id=ba80f17bd7764a0e8bcd5aebcf6500ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:34.634377 30330 tablet_service.cc:1467] Processing CreateTablet for tablet 8b637b5460b7481884ab4945ee8bd23e (DEFAULT_TABLE table=test-table [id=ba80f17bd7764a0e8bcd5aebcf6500ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:34.634721 30478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b637b5460b7481884ab4945ee8bd23e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:34.634721 30330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b637b5460b7481884ab4945ee8bd23e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:34.637344 30729 tablet_bootstrap.cc:492] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c: Bootstrap starting.
I20260810 03:40:34.637606 30628 tablet_service.cc:1467] Processing CreateTablet for tablet 8b637b5460b7481884ab4945ee8bd23e (DEFAULT_TABLE table=test-table [id=ba80f17bd7764a0e8bcd5aebcf6500ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:34.637882 30628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b637b5460b7481884ab4945ee8bd23e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:34.638453 30729 tablet_bootstrap.cc:654] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:34.639087 30728 tablet_bootstrap.cc:492] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3: Bootstrap starting.
I20260810 03:40:34.639933 30730 tablet_bootstrap.cc:492] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3: Bootstrap starting.
I20260810 03:40:34.640120 30728 tablet_bootstrap.cc:654] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:34.640793 30730 tablet_bootstrap.cc:654] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:34.641633 30729 tablet_bootstrap.cc:492] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c: No bootstrap required, opened a new log
I20260810 03:40:34.641731 30729 ts_tablet_manager.cc:1397] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 03:40:34.641945 30730 tablet_bootstrap.cc:492] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3: No bootstrap required, opened a new log
I20260810 03:40:34.642052 30730 ts_tablet_manager.cc:1397] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 03:40:34.642180 30729 raft_consensus.cc:348] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.642310 30729 raft_consensus.cc:374] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:34.642339 30729 raft_consensus.cc:729] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4962c93336104d7790c2f3e5d1ed5c8c, State: Initialized, Role: FOLLOWER
I20260810 03:40:34.642465 30729 consensus_queue.cc:260] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.642485 30730 raft_consensus.cc:348] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.642632 30730 raft_consensus.cc:374] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:34.642699 30730 raft_consensus.cc:729] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe9b28af39403f8287d5ad827007e3, State: Initialized, Role: FOLLOWER
I20260810 03:40:34.642719 30729 ts_tablet_manager.cc:1428] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 03:40:34.642839 30730 consensus_queue.cc:260] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.643090 30730 ts_tablet_manager.cc:1428] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:40:34.643240 30728 tablet_bootstrap.cc:492] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3: No bootstrap required, opened a new log
I20260810 03:40:34.643327 30728 ts_tablet_manager.cc:1397] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 03:40:34.643766 30728 raft_consensus.cc:348] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.643877 30728 raft_consensus.cc:374] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:34.643910 30728 raft_consensus.cc:729] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122fedfa36e54e0eb3a9ed06808551f3, State: Initialized, Role: FOLLOWER
I20260810 03:40:34.644025 30728 consensus_queue.cc:260] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.646229 30728 ts_tablet_manager.cc:1428] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260810 03:40:34.662314 30736 raft_consensus.cc:482] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:34.662464 30736 raft_consensus.cc:504] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.663717 30736 leader_election.cc:288] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4962c93336104d7790c2f3e5d1ed5c8c (127.21.175.66:39665), 00fe9b28af39403f8287d5ad827007e3 (127.21.175.67:39917)
I20260810 03:40:34.668692 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.664312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:57609 (local address 127.21.175.66:39665)
0810 03:40:34.664443 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:34.664448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:34.664554 (+ 106us) server_negotiation.cc:407] Connection header received
0810 03:40:34.664768 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:34.664776 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:34.664852 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:34.664931 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:34.665772 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.666360 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.667580 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.668002 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.668101 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.668185 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:34.668453 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 03:40:34.668544 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:40:34.668689 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.664151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:36367 (local address 127.21.175.67:39917)
0810 03:40:34.664345 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:34.664350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:34.664678 (+ 328us) server_negotiation.cc:407] Connection header received
0810 03:40:34.664862 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:34.664865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:34.664937 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:34.665004 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:34.665439 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.666124 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.667625 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.668032 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.668179 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.668233 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:34.668453 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 03:40:34.668552 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260810 03:40:34.668747 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.664040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:39917 (local address 127.21.175.65:36367)
0810 03:40:34.664554 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:34.664569 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:34.664697 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:34.665140 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:34.665143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:34.665164 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:34.665289 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.665296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.666252 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.666255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.667456 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.667466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.668185 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.668190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.668317 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.668343 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:40:34.668431 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 03:40:34.668564 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":280,"thread_start_us":109,"threads_started":1}
I20260810 03:40:34.668751 30806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.664040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:39665 (local address 127.21.175.65:57609)
0810 03:40:34.664432 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:34.664459 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:34.664587 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:34.664946 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:34.664949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:34.664967 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:34.665630 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.665641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.666489 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.666492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.667444 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:34.667454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.668154 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.668158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:34.668257 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.668336 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 03:40:34.668434 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 03:40:34.668585 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":97,"thread_start_us":94,"threads_started":1}
I20260810 03:40:34.669286 30499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b637b5460b7481884ab4945ee8bd23e" candidate_uuid: "122fedfa36e54e0eb3a9ed06808551f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" is_pre_election: true
I20260810 03:40:34.669400 30650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b637b5460b7481884ab4945ee8bd23e" candidate_uuid: "122fedfa36e54e0eb3a9ed06808551f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe9b28af39403f8287d5ad827007e3" is_pre_election: true
I20260810 03:40:34.669461 30499 raft_consensus.cc:2454] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 122fedfa36e54e0eb3a9ed06808551f3 in term 0.
I20260810 03:40:34.669559 30650 raft_consensus.cc:2454] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 122fedfa36e54e0eb3a9ed06808551f3 in term 0.
I20260810 03:40:34.669934 30278 leader_election.cc:302] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00fe9b28af39403f8287d5ad827007e3, 122fedfa36e54e0eb3a9ed06808551f3, 4962c93336104d7790c2f3e5d1ed5c8c; no voters:
I20260810 03:40:34.670195 30736 raft_consensus.cc:2781] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:40:34.670275 30736 raft_consensus.cc:482] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:40:34.670325 30736 raft_consensus.cc:3037] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:34.671510 30736 raft_consensus.cc:504] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.671932 30736 leader_election.cc:288] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [CANDIDATE]: Term 1 election: Requested vote from peers 4962c93336104d7790c2f3e5d1ed5c8c (127.21.175.66:39665), 00fe9b28af39403f8287d5ad827007e3 (127.21.175.67:39917)
I20260810 03:40:34.672366 30499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b637b5460b7481884ab4945ee8bd23e" candidate_uuid: "122fedfa36e54e0eb3a9ed06808551f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4962c93336104d7790c2f3e5d1ed5c8c"
I20260810 03:40:34.672366 30650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b637b5460b7481884ab4945ee8bd23e" candidate_uuid: "122fedfa36e54e0eb3a9ed06808551f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe9b28af39403f8287d5ad827007e3"
I20260810 03:40:34.672523 30499 raft_consensus.cc:3037] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:34.672592 30650 raft_consensus.cc:3037] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:34.673890 30650 raft_consensus.cc:2454] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 122fedfa36e54e0eb3a9ed06808551f3 in term 1.
I20260810 03:40:34.673890 30499 raft_consensus.cc:2454] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 122fedfa36e54e0eb3a9ed06808551f3 in term 1.
I20260810 03:40:34.674292 30278 leader_election.cc:302] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00fe9b28af39403f8287d5ad827007e3, 122fedfa36e54e0eb3a9ed06808551f3, 4962c93336104d7790c2f3e5d1ed5c8c; no voters:
I20260810 03:40:34.674475 30736 raft_consensus.cc:2781] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:34.674724 30736 raft_consensus.cc:686] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 LEADER]: Becoming Leader. State: Replica: 122fedfa36e54e0eb3a9ed06808551f3, State: Running, Role: LEADER
I20260810 03:40:34.674973 30736 consensus_queue.cc:237] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.677871 30184 catalog_manager.cc:5495] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 122fedfa36e54e0eb3a9ed06808551f3 (127.21.175.65). New cstate: current_term: 1 leader_uuid: "122fedfa36e54e0eb3a9ed06808551f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } health_report { overall_health: UNKNOWN } } }
I20260810 03:40:34.695991 30727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.691871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37432 (local address 127.21.175.65:41379)
0810 03:40:34.691997 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:34.692001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:34.692038 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:40:34.692244 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:34.692246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:34.692310 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:34.692454 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:40:34.692764 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.693261 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.694247 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.694596 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.694704 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.694965 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:40:34.695574 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:40:34.695687 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:34.695812 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 03:40:34.695876 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 03:40:34.698869 30499 raft_consensus.cc:1261] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Refusing update from remote peer 122fedfa36e54e0eb3a9ed06808551f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:34.698872 30650 raft_consensus.cc:1261] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3 [term 1 FOLLOWER]: Refusing update from remote peer 122fedfa36e54e0eb3a9ed06808551f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:34.699390 30736 consensus_queue.cc:1035] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:34.699672 30808 consensus_queue.cc:1035] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:34.712765 30186 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:34.712957 30186 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:34.715446 30186 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c: Sending DeleteTablet for 3 replicas of tablet 8b637b5460b7481884ab4945ee8bd23e
I20260810 03:40:34.716151 30330 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b637b5460b7481884ab4945ee8bd23e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:34 UTC) from {username='oryx', principal='oryx/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:37424
I20260810 03:40:34.716163 30478 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b637b5460b7481884ab4945ee8bd23e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:34 UTC) from {username='oryx', principal='oryx/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:59068
I20260810 03:40:34.716359 30752 tablet_replica.cc:330] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c: stopping tablet replica
I20260810 03:40:34.716361 30754 tablet_replica.cc:330] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3: stopping tablet replica
I20260810 03:40:34.716476 30754 raft_consensus.cc:2229] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:40:34.716480 30752 raft_consensus.cc:2229] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:34.716640 30752 raft_consensus.cc:2258] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:34.716820 30754 raft_consensus.cc:2258] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:34.717327 30628 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b637b5460b7481884ab4945ee8bd23e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:34 UTC) from {username='oryx', principal='oryx/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:50950
I20260810 03:40:34.717569 30753 tablet_replica.cc:330] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3: stopping tablet replica
I20260810 03:40:34.717649 30752 ts_tablet_manager.cc:1905] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:34.717463 30184 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 03:40:34.717697 30753 raft_consensus.cc:2229] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:34.717833 30753 raft_consensus.cc:2258] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 03:40:34.718056 30184 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:40:34.718571 30754 ts_tablet_manager.cc:1905] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:34.720338 30752 ts_tablet_manager.cc:1918] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:40:34.720409 30752 log.cc:1196] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/8b637b5460b7481884ab4945ee8bd23e
I20260810 03:40:34.720672 30752 ts_tablet_manager.cc:1939] T 8b637b5460b7481884ab4945ee8bd23e P 4962c93336104d7790c2f3e5d1ed5c8c: Deleting consensus metadata
I20260810 03:40:34.721303 30754 ts_tablet_manager.cc:1918] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:40:34.721387 30754 log.cc:1196] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-0/wal/wals/8b637b5460b7481884ab4945ee8bd23e
I20260810 03:40:34.721987 30167 catalog_manager.cc:4841] TS 4962c93336104d7790c2f3e5d1ed5c8c (127.21.175.66:39665): tablet 8b637b5460b7481884ab4945ee8bd23e (table test-table [id=ba80f17bd7764a0e8bcd5aebcf6500ec]) successfully deleted
I20260810 03:40:34.722222 30754 ts_tablet_manager.cc:1939] T 8b637b5460b7481884ab4945ee8bd23e P 122fedfa36e54e0eb3a9ed06808551f3: Deleting consensus metadata
I20260810 03:40:34.723431 30753 ts_tablet_manager.cc:1905] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:34.723752 30478 tablet_service.cc:1467] Processing CreateTablet for tablet b689df491d7c45de862e1a41f28a6dcd (DEFAULT_TABLE table=test-table [id=6be9d3f7cb3b448ca38d20ee0f920ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:34.723875 30628 tablet_service.cc:1467] Processing CreateTablet for tablet b689df491d7c45de862e1a41f28a6dcd (DEFAULT_TABLE table=test-table [id=6be9d3f7cb3b448ca38d20ee0f920ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:34.724009 30478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b689df491d7c45de862e1a41f28a6dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:34.724150 30628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b689df491d7c45de862e1a41f28a6dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:34.726116 30753 ts_tablet_manager.cc:1918] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:40:34.726197 30753 log.cc:1196] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/8b637b5460b7481884ab4945ee8bd23e
I20260810 03:40:34.726473 30730 tablet_bootstrap.cc:492] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3: Bootstrap starting.
I20260810 03:40:34.726503 30729 tablet_bootstrap.cc:492] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c: Bootstrap starting.
I20260810 03:40:34.727332 30729 tablet_bootstrap.cc:654] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:34.727332 30730 tablet_bootstrap.cc:654] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:34.727339 30330 tablet_service.cc:1467] Processing CreateTablet for tablet b689df491d7c45de862e1a41f28a6dcd (DEFAULT_TABLE table=test-table [id=6be9d3f7cb3b448ca38d20ee0f920ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:34.727602 30330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b689df491d7c45de862e1a41f28a6dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:34.728188 30753 ts_tablet_manager.cc:1939] T 8b637b5460b7481884ab4945ee8bd23e P 00fe9b28af39403f8287d5ad827007e3: Deleting consensus metadata
I20260810 03:40:34.728819 30729 tablet_bootstrap.cc:492] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c: No bootstrap required, opened a new log
I20260810 03:40:34.728912 30729 ts_tablet_manager.cc:1397] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 03:40:34.729045 30168 catalog_manager.cc:4841] TS 00fe9b28af39403f8287d5ad827007e3 (127.21.175.67:39917): tablet 8b637b5460b7481884ab4945ee8bd23e (table test-table [id=ba80f17bd7764a0e8bcd5aebcf6500ec]) successfully deleted
I20260810 03:40:34.729373 30729 raft_consensus.cc:348] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.729534 30729 raft_consensus.cc:374] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:34.729580 30729 raft_consensus.cc:729] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4962c93336104d7790c2f3e5d1ed5c8c, State: Initialized, Role: FOLLOWER
I20260810 03:40:34.729732 30729 consensus_queue.cc:260] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.729959 30728 tablet_bootstrap.cc:492] T b689df491d7c45de862e1a41f28a6dcd P 122fedfa36e54e0eb3a9ed06808551f3: Bootstrap starting.
I20260810 03:40:34.730036 30729 ts_tablet_manager.cc:1428] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:40:34.730129 30730 tablet_bootstrap.cc:492] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3: No bootstrap required, opened a new log
I20260810 03:40:34.730204 30730 ts_tablet_manager.cc:1397] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 03:40:34.730535 30730 raft_consensus.cc:348] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.730628 30730 raft_consensus.cc:374] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:34.730648 30730 raft_consensus.cc:729] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe9b28af39403f8287d5ad827007e3, State: Initialized, Role: FOLLOWER
I20260810 03:40:34.730784 30730 consensus_queue.cc:260] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.730907 30728 tablet_bootstrap.cc:654] T b689df491d7c45de862e1a41f28a6dcd P 122fedfa36e54e0eb3a9ed06808551f3: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:34.731058 30730 ts_tablet_manager.cc:1428] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 03:40:34.731921 30169 catalog_manager.cc:4841] TS 122fedfa36e54e0eb3a9ed06808551f3 (127.21.175.65:41379): tablet 8b637b5460b7481884ab4945ee8bd23e (table test-table [id=ba80f17bd7764a0e8bcd5aebcf6500ec]) successfully deleted
I20260810 03:40:34.732256 30728 tablet_bootstrap.cc:492] T b689df491d7c45de862e1a41f28a6dcd P 122fedfa36e54e0eb3a9ed06808551f3: No bootstrap required, opened a new log
I20260810 03:40:34.732359 30728 ts_tablet_manager.cc:1397] T b689df491d7c45de862e1a41f28a6dcd P 122fedfa36e54e0eb3a9ed06808551f3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:40:34.732838 30728 raft_consensus.cc:348] T b689df491d7c45de862e1a41f28a6dcd P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.732951 30728 raft_consensus.cc:374] T b689df491d7c45de862e1a41f28a6dcd P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:34.732985 30728 raft_consensus.cc:729] T b689df491d7c45de862e1a41f28a6dcd P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122fedfa36e54e0eb3a9ed06808551f3, State: Initialized, Role: FOLLOWER
I20260810 03:40:34.733109 30728 consensus_queue.cc:260] T b689df491d7c45de862e1a41f28a6dcd P 122fedfa36e54e0eb3a9ed06808551f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.733368 30728 ts_tablet_manager.cc:1428] T b689df491d7c45de862e1a41f28a6dcd P 122fedfa36e54e0eb3a9ed06808551f3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:40:34.769170 30738 raft_consensus.cc:482] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:34.769345 30738 raft_consensus.cc:504] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.769856 30738 leader_election.cc:288] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 122fedfa36e54e0eb3a9ed06808551f3 (127.21.175.65:41379), 4962c93336104d7790c2f3e5d1ed5c8c (127.21.175.66:39665)
I20260810 03:40:34.770290 30499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b689df491d7c45de862e1a41f28a6dcd" candidate_uuid: "00fe9b28af39403f8287d5ad827007e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" is_pre_election: true
I20260810 03:40:34.770292 30350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b689df491d7c45de862e1a41f28a6dcd" candidate_uuid: "00fe9b28af39403f8287d5ad827007e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122fedfa36e54e0eb3a9ed06808551f3" is_pre_election: true
I20260810 03:40:34.770464 30499 raft_consensus.cc:2454] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe9b28af39403f8287d5ad827007e3 in term 0.
I20260810 03:40:34.770462 30350 raft_consensus.cc:2454] T b689df491d7c45de862e1a41f28a6dcd P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe9b28af39403f8287d5ad827007e3 in term 0.
I20260810 03:40:34.770809 30574 leader_election.cc:302] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe9b28af39403f8287d5ad827007e3, 4962c93336104d7790c2f3e5d1ed5c8c; no voters:
I20260810 03:40:34.771004 30738 raft_consensus.cc:2781] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:40:34.771090 30738 raft_consensus.cc:482] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:40:34.771152 30738 raft_consensus.cc:3037] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:34.772189 30738 raft_consensus.cc:504] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.772564 30738 leader_election.cc:288] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [CANDIDATE]: Term 1 election: Requested vote from peers 122fedfa36e54e0eb3a9ed06808551f3 (127.21.175.65:41379), 4962c93336104d7790c2f3e5d1ed5c8c (127.21.175.66:39665)
I20260810 03:40:34.772974 30350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b689df491d7c45de862e1a41f28a6dcd" candidate_uuid: "00fe9b28af39403f8287d5ad827007e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122fedfa36e54e0eb3a9ed06808551f3"
I20260810 03:40:34.772957 30499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b689df491d7c45de862e1a41f28a6dcd" candidate_uuid: "00fe9b28af39403f8287d5ad827007e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4962c93336104d7790c2f3e5d1ed5c8c"
I20260810 03:40:34.773118 30499 raft_consensus.cc:3037] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:34.773128 30350 raft_consensus.cc:3037] T b689df491d7c45de862e1a41f28a6dcd P 122fedfa36e54e0eb3a9ed06808551f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:34.774183 30350 raft_consensus.cc:2454] T b689df491d7c45de862e1a41f28a6dcd P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe9b28af39403f8287d5ad827007e3 in term 1.
I20260810 03:40:34.774333 30499 raft_consensus.cc:2454] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe9b28af39403f8287d5ad827007e3 in term 1.
I20260810 03:40:34.774565 30577 leader_election.cc:302] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe9b28af39403f8287d5ad827007e3, 122fedfa36e54e0eb3a9ed06808551f3; no voters:
I20260810 03:40:34.774789 30738 raft_consensus.cc:2781] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:34.774887 30738 raft_consensus.cc:686] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [term 1 LEADER]: Becoming Leader. State: Replica: 00fe9b28af39403f8287d5ad827007e3, State: Running, Role: LEADER
I20260810 03:40:34.775039 30738 consensus_queue.cc:237] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } }
I20260810 03:40:34.776932 30184 catalog_manager.cc:5495] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe9b28af39403f8287d5ad827007e3 (127.21.175.67). New cstate: current_term: 1 leader_uuid: "00fe9b28af39403f8287d5ad827007e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00fe9b28af39403f8287d5ad827007e3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39917 } health_report { overall_health: HEALTHY } } }
I20260810 03:40:34.802068 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.798266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.21.175.67:39917)
0810 03:40:34.798430 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:34.798437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:34.798454 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:40:34.798506 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:34.798509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:34.798564 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:34.798684 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:40:34.798989 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.799610 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.800724 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:34.800990 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:34.801148 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:34.801445 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:40:34.801607 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:40:34.801722 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:34.801833 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 03:40:34.801895 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:40:34.809150 30499 raft_consensus.cc:1261] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Refusing update from remote peer 00fe9b28af39403f8287d5ad827007e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:34.809643 30738 consensus_queue.cc:1035] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4962c93336104d7790c2f3e5d1ed5c8c" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:34.809846 30350 raft_consensus.cc:1261] T b689df491d7c45de862e1a41f28a6dcd P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 FOLLOWER]: Refusing update from remote peer 00fe9b28af39403f8287d5ad827007e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:34.810235 30734 consensus_queue.cc:1035] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "122fedfa36e54e0eb3a9ed06808551f3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:34.856804 30184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:34.856999 30184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:34.859555 30184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4e77b4104c04dadaeea97b78db5924c: Sending DeleteTablet for 3 replicas of tablet b689df491d7c45de862e1a41f28a6dcd
I20260810 03:40:34.860260 30478 tablet_service.cc:1514] Processing DeleteTablet for tablet b689df491d7c45de862e1a41f28a6dcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:34 UTC) from {username='oryx', principal='oryx/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:59068
I20260810 03:40:34.860294 30330 tablet_service.cc:1514] Processing DeleteTablet for tablet b689df491d7c45de862e1a41f28a6dcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:34 UTC) from {username='oryx', principal='oryx/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:37424
I20260810 03:40:34.860500 30752 tablet_replica.cc:330] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c: stopping tablet replica
I20260810 03:40:34.860517 30754 tablet_replica.cc:330] T b689df491d7c45de862e1a41f28a6dcd P 122fedfa36e54e0eb3a9ed06808551f3: stopping tablet replica
I20260810 03:40:34.860635 30754 raft_consensus.cc:2229] T b689df491d7c45de862e1a41f28a6dcd P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:34.860652 30752 raft_consensus.cc:2229] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:34.860785 30754 raft_consensus.cc:2258] T b689df491d7c45de862e1a41f28a6dcd P 122fedfa36e54e0eb3a9ed06808551f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:34.860803 30752 raft_consensus.cc:2258] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:34.861572 30628 tablet_service.cc:1514] Processing DeleteTablet for tablet b689df491d7c45de862e1a41f28a6dcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:34 UTC) from {username='oryx', principal='oryx/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:50950
I20260810 03:40:34.861768 30753 tablet_replica.cc:330] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3: stopping tablet replica
I20260810 03:40:34.861748 30752 ts_tablet_manager.cc:1905] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:34.861866 30753 raft_consensus.cc:2229] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:40:34.862073 30753 raft_consensus.cc:2258] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:34.864140 30752 ts_tablet_manager.cc:1918] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:40:34.864204 30752 log.cc:1196] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/b689df491d7c45de862e1a41f28a6dcd
I20260810 03:40:34.864393 30752 ts_tablet_manager.cc:1939] T b689df491d7c45de862e1a41f28a6dcd P 4962c93336104d7790c2f3e5d1ed5c8c: Deleting consensus metadata
I20260810 03:40:34.865334 30167 catalog_manager.cc:4841] TS 4962c93336104d7790c2f3e5d1ed5c8c (127.21.175.66:39665): tablet b689df491d7c45de862e1a41f28a6dcd (table test-table [id=6be9d3f7cb3b448ca38d20ee0f920ae6]) successfully deleted
I20260810 03:40:34.865585 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 30259
I20260810 03:40:34.868422 30753 ts_tablet_manager.cc:1905] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:34.871047 30753 ts_tablet_manager.cc:1918] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:40:34.871140 30753 log.cc:1196] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/b689df491d7c45de862e1a41f28a6dcd
I20260810 03:40:34.871393 30753 ts_tablet_manager.cc:1939] T b689df491d7c45de862e1a41f28a6dcd P 00fe9b28af39403f8287d5ad827007e3: Deleting consensus metadata
I20260810 03:40:34.872218 30168 catalog_manager.cc:4841] TS 00fe9b28af39403f8287d5ad827007e3 (127.21.175.67:39917): tablet b689df491d7c45de862e1a41f28a6dcd (table test-table [id=6be9d3f7cb3b448ca38d20ee0f920ae6]) successfully deleted
W20260810 03:40:34.877256 30169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.175.65:41379 (error 108)
I20260810 03:40:34.877389 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 30408
I20260810 03:40:34.880692 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:34.880219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:41379 (local address 127.0.0.1:37442)
0810 03:40:34.880496 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:34.880584 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:41379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
W20260810 03:40:34.880844 30169 catalog_manager.cc:4584] TS 122fedfa36e54e0eb3a9ed06808551f3 (127.21.175.65:41379): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b689df491d7c45de862e1a41f28a6dcd: Network error: Client connection negotiation failed: client connection to 127.21.175.65:41379: connect: Connection refused (error 111)
I20260810 03:40:34.887745 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 30557
I20260810 03:40:34.896749 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 30150
2026-08-10T03:40:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3330 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:40:34 dist-test-slave-ss4n krb5kdc[30826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:40:34 dist-test-slave-ss4n krb5kdc[30826](info): set up 2 sockets
Aug 10 03:40:34 dist-test-slave-ss4n krb5kdc[30826](info): commencing operation
krb5kdc: starting...
W20260810 03:40:36.002259 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:40:36 dist-test-slave-ss4n krb5kdc[30826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:40:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:40:36.145319 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:36089
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:37157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:36089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:36.266880 30839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:36.267185 30839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:36.267226 30839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:36.271515 30839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:36.271597 30839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:36.271617 30839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:36.271639 30839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:36.271706 30839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:40:36.277263 30839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:37157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:36089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:36089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:36.278518 30839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:36.279579 30839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:36.288929 30848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:36.289022 30839 server_base.cc:1034] running on GCE node
W20260810 03:40:36.288929 30846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:36.288923 30845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:36.290023 30839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:36.291325 30839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:36.292533 30839 hybrid_clock.cc:648] HybridClock initialized: now 1786333236292505 us; error 51 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T03:40:36Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1404 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:40:36 dist-test-slave-ss4n krb5kdc[30853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:40:36 dist-test-slave-ss4n krb5kdc[30853](info): set up 2 sockets
Aug 10 03:40:36 dist-test-slave-ss4n krb5kdc[30853](info): commencing operation
krb5kdc: starting...
W20260810 03:40:37.401973 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:40:37 dist-test-slave-ss4n krb5kdc[30853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:40:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:40:37.548591 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:46473
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:46473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:37.669409 30872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:37.669754 30872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:37.669796 30872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:37.673967 30872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:37.674044 30872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:37.674063 30872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:37.674081 30872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:37.674096 30872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:40:37.680547 30872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:46473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:46473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:37.681833 30872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:37.682945 30872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:37.692193 30879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:37.692186 30878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:37.692250 30872 server_base.cc:1034] running on GCE node
W20260810 03:40:37.692186 30881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:37.693081 30872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:37.694393 30872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:37.695616 30872 hybrid_clock.cc:648] HybridClock initialized: now 1786333237695592 us; error 36 us; skew 500 ppm
Aug 10 03:40:37 dist-test-slave-ss4n krb5kdc[30853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333237, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:37.698643 30872 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:40:37.699625 30872 webserver.cc:460] Webserver started at http://127.21.175.126:44365/ using document root <none> and password file <none>
I20260810 03:40:37.700255 30872 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:37.700357 30872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:37.700601 30872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:37.702898 30872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "41131c3d33f544f59d19c2c84fe8d08e"
format_stamp: "Formatted at 2026-08-10 03:40:37 on dist-test-slave-ss4n"
server_key: "3aa636320e16d59bbb26e87f91287a6e"
server_key_iv: "7868f42c09d4f64d562c3b95d3d0d5ca"
server_key_version: "encryptionkey@0"
I20260810 03:40:37.703518 30872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "41131c3d33f544f59d19c2c84fe8d08e"
format_stamp: "Formatted at 2026-08-10 03:40:37 on dist-test-slave-ss4n"
server_key: "3aa636320e16d59bbb26e87f91287a6e"
server_key_iv: "7868f42c09d4f64d562c3b95d3d0d5ca"
server_key_version: "encryptionkey@0"
I20260810 03:40:37.707469 30872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:40:37.710251 30889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:37.711637 30872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:37.711792 30872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "41131c3d33f544f59d19c2c84fe8d08e"
format_stamp: "Formatted at 2026-08-10 03:40:37 on dist-test-slave-ss4n"
server_key: "3aa636320e16d59bbb26e87f91287a6e"
server_key_iv: "7868f42c09d4f64d562c3b95d3d0d5ca"
server_key_version: "encryptionkey@0"
I20260810 03:40:37.711920 30872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:37.724751 30872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:37.728471 30872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:37.728703 30872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:37.736495 30872 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:46473
I20260810 03:40:37.736567 30944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:46473 every 8 connection(s)
I20260810 03:40:37.737617 30872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:37.740679 30945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:37.747004 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 30872
I20260810 03:40:37.747040 30945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e: Bootstrap starting.
I20260810 03:40:37.747177 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:40:37.747522 22205 external_mini_cluster.cc:1428] Setting key 108c1c18243cffb1910cc255bb025044
I20260810 03:40:37.750079 30945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:37.750913 30945 log.cc:824] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:37.753217 30945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e: No bootstrap required, opened a new log
Aug 10 03:40:37 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:37.758112 30945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41131c3d33f544f59d19c2c84fe8d08e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46473 } }
I20260810 03:40:37.758469 30945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:37.758567 30945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41131c3d33f544f59d19c2c84fe8d08e, State: Initialized, Role: FOLLOWER
I20260810 03:40:37.759085 30945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41131c3d33f544f59d19c2c84fe8d08e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46473 } }
I20260810 03:40:37.759240 30945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:37.759306 30945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:37.759418 30945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:37.760733 30945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41131c3d33f544f59d19c2c84fe8d08e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46473 } }
I20260810 03:40:37.761219 30945 leader_election.cc:302] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41131c3d33f544f59d19c2c84fe8d08e; no voters:
I20260810 03:40:37.761729 30945 leader_election.cc:288] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:40:37.761852 30951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:37.762130 30951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [term 1 LEADER]: Becoming Leader. State: Replica: 41131c3d33f544f59d19c2c84fe8d08e, State: Running, Role: LEADER
I20260810 03:40:37.762563 30951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41131c3d33f544f59d19c2c84fe8d08e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46473 } }
I20260810 03:40:37.763465 30945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:37.763940 30951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41131c3d33f544f59d19c2c84fe8d08e. Latest consensus state: current_term: 1 leader_uuid: "41131c3d33f544f59d19c2c84fe8d08e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41131c3d33f544f59d19c2c84fe8d08e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46473 } } }
I20260810 03:40:37.764715 30951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:37.764762 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:37.749035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33858 (local address 127.21.175.126:46473)
0810 03:40:37.749518 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:37.749536 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:37.749575 (+ 39us) server_negotiation.cc:407] Connection header received
0810 03:40:37.750365 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:37.750387 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:37.750766 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:37.751187 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:37.752035 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:37.753062 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:37.754282 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:37.754798 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:37.754929 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:37.758054 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:37.758061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:37.758080 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:37.758123 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:37.760191 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:37.760964 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:37.760970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:37.760979 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:37.761142 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:37.761550 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:37.761553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:37.761554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:37.762190 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:37.763154 (+ 964us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:37.763317 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 03:40:37.763653 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":149,"threads_started":1}
I20260810 03:40:37.764605 30952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41131c3d33f544f59d19c2c84fe8d08e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41131c3d33f544f59d19c2c84fe8d08e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46473 } } }
I20260810 03:40:37.765450 30952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:37.765200 30962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:37.768821 30962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:37.775604 30962 catalog_manager.cc:1347] Generated new cluster ID: d1e30ecf25a94aa884b285eabd06a868
I20260810 03:40:37.775699 30962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:37.781270 30962 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:40:37.782351 30962 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:37.803462 30962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e: Generated new TSK 0
I20260810 03:40:37.804297 30962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:40:37.854120 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46473
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:38.000900 30974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:38.001219 30974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:38.001266 30974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:38.005558 30974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:38.005631 30974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:38.005694 30974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:40:38.011186 30974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:38.012323 30974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:38.013464 30974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:38.023976 30983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:38.023979 30980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:38.023918 30981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:38.024288 30974 server_base.cc:1034] running on GCE node
I20260810 03:40:38.024760 30974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:38.025455 30974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:38.026726 30974 hybrid_clock.cc:648] HybridClock initialized: now 1786333238026689 us; error 48 us; skew 500 ppm
Aug 10 03:40:38 dist-test-slave-ss4n krb5kdc[30853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333238, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:38.029973 30974 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:40:38.030943 30974 webserver.cc:460] Webserver started at http://127.21.175.65:45821/ using document root <none> and password file <none>
I20260810 03:40:38.031498 30974 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:38.031562 30974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:38.031745 30974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:38.034000 30974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "4efa09cf84a045468a88d118ea07a5cb"
format_stamp: "Formatted at 2026-08-10 03:40:38 on dist-test-slave-ss4n"
server_key: "853b6f8048a55ff3cd924ddf269f5f1c"
server_key_iv: "381bc83a9bd0d98cedc271a5999c4e63"
server_key_version: "encryptionkey@0"
I20260810 03:40:38.034564 30974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "4efa09cf84a045468a88d118ea07a5cb"
format_stamp: "Formatted at 2026-08-10 03:40:38 on dist-test-slave-ss4n"
server_key: "853b6f8048a55ff3cd924ddf269f5f1c"
server_key_iv: "381bc83a9bd0d98cedc271a5999c4e63"
server_key_version: "encryptionkey@0"
I20260810 03:40:38.038878 30974 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:40:38.041924 30990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:38.043452 30974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:38.043566 30974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "4efa09cf84a045468a88d118ea07a5cb"
format_stamp: "Formatted at 2026-08-10 03:40:38 on dist-test-slave-ss4n"
server_key: "853b6f8048a55ff3cd924ddf269f5f1c"
server_key_iv: "381bc83a9bd0d98cedc271a5999c4e63"
server_key_version: "encryptionkey@0"
I20260810 03:40:38.043682 30974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:38.062376 30974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:38.066258 30974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:38.066428 30974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:38.067018 30974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:38.067921 30974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:38.067996 30974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:38.068074 30974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:38.068126 30974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:38.077394 30974 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:35555
I20260810 03:40:38.077418 31116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:35555 every 8 connection(s)
I20260810 03:40:38.078454 30974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:40:38.084393 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 30974
I20260810 03:40:38.084547 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:40:38.084856 22205 external_mini_cluster.cc:1428] Setting key af1145aa628f75d9e7b867f50cb57536
Aug 10 03:40:38 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333238, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:38.094957 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:38.080692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:53169 (local address 127.21.175.126:46473)
0810 03:40:38.080866 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:38.080871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:38.082126 (+ 1255us) server_negotiation.cc:407] Connection header received
0810 03:40:38.083001 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:38.083004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:38.083051 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:38.083151 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:38.084685 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:38.085306 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:38.086928 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:38.087166 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:38.087253 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:38.090628 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:38.090631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:38.090633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:38.090659 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:38.092356 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:38.093097 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:38.093100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:38.093108 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:38.093195 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:38.093683 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:38.093686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:38.093687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:38.093884 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:38.093961 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:38.094704 (+ 743us) server_negotiation.cc:299] Negotiation successful
0810 03:40:38.094824 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 03:40:38.096318 31119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:38.081074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:53169)
0810 03:40:38.081945 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:38.081991 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:38.082806 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:38.083307 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:38.083314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:38.083756 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:38.084458 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:38.084481 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:38.085478 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:38.085482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:38.086788 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:38.086800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:38.087338 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:38.087345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:38.087622 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:38.088365 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:38.088393 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:38.090364 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:38.092497 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:38.092506 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:38.092530 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:38.092954 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:38.093300 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:38.093305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:38.093310 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:38.093571 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:38.093998 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:38.094009 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:38.094411 (+ 402us) client_negotiation.cc:769] Sending connection context
0810 03:40:38.094655 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 03:40:38.095086 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":143,"threads_started":1}
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
I20260810 03:40:38.098166 31117 heartbeater.cc:347] Connected to a master server at 127.21.175.126:46473
I20260810 03:40:38.098434 31117 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:38.099004 31117 heartbeater.cc:511] Master 127.21.175.126:46473 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
I20260810 03:40:38.100894 30905 ts_manager.cc:194] Registered new tserver with Master: 4efa09cf84a045468a88d118ea07a5cb (127.21.175.65:35555)
I20260810 03:40:38.102173 30905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:53169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:40:38.122028 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46473
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:38.241940 31122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:38.242308 31122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:38.242390 31122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:38.246707 31122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:38.246829 31122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:38.246942 31122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:40:38.252672 31122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:38.253894 31122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:38.254984 31122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:38.265457 31128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:38.265501 31127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:38.265640 31122 server_base.cc:1034] running on GCE node
W20260810 03:40:38.265825 31130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:38.266304 31122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:38.267004 31122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:38.268204 31122 hybrid_clock.cc:648] HybridClock initialized: now 1786333238268162 us; error 87 us; skew 500 ppm
Aug 10 03:40:38 dist-test-slave-ss4n krb5kdc[30853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333238, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:38.271410 31122 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:40:38.272315 31122 webserver.cc:460] Webserver started at http://127.21.175.66:44317/ using document root <none> and password file <none>
I20260810 03:40:38.272842 31122 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:38.272902 31122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:38.273072 31122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:38.275270 31122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "00deef55fa4c4caa90576316b7cf2682"
format_stamp: "Formatted at 2026-08-10 03:40:38 on dist-test-slave-ss4n"
server_key: "65165154f65d104e0680213c54fbd772"
server_key_iv: "fcb22c899bf7c65abd91a3556ad40d75"
server_key_version: "encryptionkey@0"
I20260810 03:40:38.275799 31122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "00deef55fa4c4caa90576316b7cf2682"
format_stamp: "Formatted at 2026-08-10 03:40:38 on dist-test-slave-ss4n"
server_key: "65165154f65d104e0680213c54fbd772"
server_key_iv: "fcb22c899bf7c65abd91a3556ad40d75"
server_key_version: "encryptionkey@0"
I20260810 03:40:38.279752 31122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 03:40:38.282677 31137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:38.284262 31122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:40:38.284396 31122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "00deef55fa4c4caa90576316b7cf2682"
format_stamp: "Formatted at 2026-08-10 03:40:38 on dist-test-slave-ss4n"
server_key: "65165154f65d104e0680213c54fbd772"
server_key_iv: "fcb22c899bf7c65abd91a3556ad40d75"
server_key_version: "encryptionkey@0"
I20260810 03:40:38.284505 31122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:38.300146 31122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:38.303736 31122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:38.303917 31122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:38.304476 31122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:38.305287 31122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:38.305335 31122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:38.305374 31122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:38.305390 31122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:38.315024 31122 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:38193
I20260810 03:40:38.315129 31260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:38193 every 8 connection(s)
I20260810 03:40:38.316138 31122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:40:38.319931 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 31122
I20260810 03:40:38.320072 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:40:38.320369 22205 external_mini_cluster.cc:1428] Setting key 4f3c7b7edc773a642caa0b167ed1fd58
Aug 10 03:40:38 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333238, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
I20260810 03:40:38.332571 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:38.318354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:54713 (local address 127.21.175.126:46473)
0810 03:40:38.318608 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:38.318612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:38.319716 (+ 1104us) server_negotiation.cc:407] Connection header received
0810 03:40:38.320860 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:38.320893 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:38.321000 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:38.321106 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:38.323048 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:38.323585 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:38.325402 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:38.325736 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:38.325828 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:38.328623 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:38.328627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:38.328630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:38.328666 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:38.330227 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:38.330794 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:38.330797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:38.330799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:38.330887 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:38.331275 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:38.331280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:38.331282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:38.331527 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:38.331642 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:38.332233 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 03:40:38.332373 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 03:40:38.333642 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:38.318775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:54713)
0810 03:40:38.319535 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:38.319592 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:38.320592 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:38.321975 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:38.321983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:38.322389 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:38.322902 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:38.322916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:38.323718 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:38.323722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:38.325253 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:38.325268 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:38.325888 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:38.325891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:38.326096 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:38.326818 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:38.326836 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:38.328437 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:38.330361 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:38.330368 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:38.330386 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:38.330681 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:38.330983 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:38.330986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:38.330988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:38.331171 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:38.331644 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:38.331653 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:38.331938 (+ 285us) client_negotiation.cc:769] Sending connection context
0810 03:40:38.332154 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 03:40:38.332523 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":144,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
I20260810 03:40:38.335031 31261 heartbeater.cc:347] Connected to a master server at 127.21.175.126:46473
I20260810 03:40:38.335330 31261 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:38.335878 31261 heartbeater.cc:511] Master 127.21.175.126:46473 requested a full tablet report, sending...
I20260810 03:40:38.337199 30905 ts_manager.cc:194] Registered new tserver with Master: 00deef55fa4c4caa90576316b7cf2682 (127.21.175.66:38193)
I20260810 03:40:38.337914 30905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:54713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:40:38.357133 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46473
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:38.475937 31266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:38.476295 31266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:38.476377 31266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:38.480571 31266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:38.480690 31266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:38.480789 31266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:40:38.486431 31266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:38.487622 31266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:38.488755 31266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:38.499078 31271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:38.499137 31272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:38.499282 31266 server_base.cc:1034] running on GCE node
W20260810 03:40:38.499095 31274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:38.499857 31266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:38.500641 31266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:38.501847 31266 hybrid_clock.cc:648] HybridClock initialized: now 1786333238501829 us; error 31 us; skew 500 ppm
Aug 10 03:40:38 dist-test-slave-ss4n krb5kdc[30853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333238, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:38.505180 31266 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:40:38.506171 31266 webserver.cc:460] Webserver started at http://127.21.175.67:38109/ using document root <none> and password file <none>
I20260810 03:40:38.506793 31266 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:38.506856 31266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:38.507153 31266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:38.509320 31266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "0d5420a5ba3e4e0d884bde254fd48e1f"
format_stamp: "Formatted at 2026-08-10 03:40:38 on dist-test-slave-ss4n"
server_key: "26b5a1f1dcd8dc0debba49b1d4c19ee8"
server_key_iv: "e81cec4d3135536c5db8d1e1682cfaff"
server_key_version: "encryptionkey@0"
I20260810 03:40:38.509974 31266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "0d5420a5ba3e4e0d884bde254fd48e1f"
format_stamp: "Formatted at 2026-08-10 03:40:38 on dist-test-slave-ss4n"
server_key: "26b5a1f1dcd8dc0debba49b1d4c19ee8"
server_key_iv: "e81cec4d3135536c5db8d1e1682cfaff"
server_key_version: "encryptionkey@0"
I20260810 03:40:38.513957 31266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:40:38.516696 31281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:38.518352 31266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:38.518457 31266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "0d5420a5ba3e4e0d884bde254fd48e1f"
format_stamp: "Formatted at 2026-08-10 03:40:38 on dist-test-slave-ss4n"
server_key: "26b5a1f1dcd8dc0debba49b1d4c19ee8"
server_key_iv: "e81cec4d3135536c5db8d1e1682cfaff"
server_key_version: "encryptionkey@0"
I20260810 03:40:38.518600 31266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:38.555298 31266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:38.558775 31266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:38.559005 31266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:38.559679 31266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:38.560582 31266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:38.560634 31266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:38.560678 31266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:38.560730 31266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:38.570546 31266 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:46551
I20260810 03:40:38.570607 31400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:46551 every 8 connection(s)
I20260810 03:40:38.571537 31266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:40:38.575484 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 31266
I20260810 03:40:38.575631 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:40:38.575981 22205 external_mini_cluster.cc:1428] Setting key 0c9f8bdbf6f2f627c190639bfeebb4c2
Aug 10 03:40:38 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333238, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:38.585723 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:38.573631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:45025 (local address 127.21.175.126:46473)
0810 03:40:38.573809 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:38.573813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:38.574778 (+ 965us) server_negotiation.cc:407] Connection header received
0810 03:40:38.575769 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:38.575775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:38.575839 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:38.575943 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:38.577342 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:38.578125 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:38.579293 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:38.579534 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:38.579686 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:38.582219 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:38.582223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:38.582226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:38.582254 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:38.583547 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:38.584119 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:38.584122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:38.584124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:38.584215 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:38.584558 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:38.584562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:38.584564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:38.584768 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:38.584877 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:38.585402 (+ 525us) server_negotiation.cc:299] Negotiation successful
0810 03:40:38.585551 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:40:38.586560 31404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:38.573953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:45025)
0810 03:40:38.574627 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:38.574669 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:38.575498 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:38.576154 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:38.576160 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:38.576565 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:38.577196 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:38.577212 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:38.578262 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:38.578265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:38.579153 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:38.579162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:38.579705 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:38.579709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:38.579900 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:38.580491 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:38.580507 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:38.582032 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:38.583718 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:38.583724 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:38.583741 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:38.584007 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:38.584309 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:38.584313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:38.584315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:38.584459 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:38.584875 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:38.584882 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:38.585150 (+ 268us) client_negotiation.cc:769] Sending connection context
0810 03:40:38.585348 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 03:40:38.585722 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":116,"threads_started":1}
I20260810 03:40:38.587841 31401 heartbeater.cc:347] Connected to a master server at 127.21.175.126:46473
I20260810 03:40:38.588104 31401 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:38.588850 31401 heartbeater.cc:511] Master 127.21.175.126:46473 requested a full tablet report, sending...
I20260810 03:40:38.590062 30906 ts_manager.cc:194] Registered new tserver with Master: 0d5420a5ba3e4e0d884bde254fd48e1f (127.21.175.67:46551)
I20260810 03:40:38.590682 30906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:45025
I20260810 03:40:38.602254 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.65.keytab.
Entry for principal oryx/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.65.keytab.
WARNING: no policy specified for oryx/127.21.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.66.keytab.
Entry for principal oryx/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.66.keytab.
WARNING: no policy specified for oryx/127.21.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.67.keytab.
Entry for principal oryx/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.67.keytab.
I20260810 03:40:38.696341 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 30974
I20260810 03:40:38.705114 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 31122
I20260810 03:40:38.713058 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 31266
I20260810 03:40:38.721350 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 30872
I20260810 03:40:38.730840 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:46473
--webserver_interface=127.21.175.126
--webserver_port=44365
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:46473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:38.854027 31412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:38.854383 31412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:38.854496 31412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:38.858685 31412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:38.858794 31412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:38.858829 31412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:38.858871 31412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:38.858906 31412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:40:38.864603 31412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:46473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:46473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=44365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:38.865914 31412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:38.867018 31412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:38.876282 31418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:38.876324 31417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:38.876324 31420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:38.876575 31412 server_base.cc:1034] running on GCE node
I20260810 03:40:38.877430 31412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:38.878818 31412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:38.880040 31412 hybrid_clock.cc:648] HybridClock initialized: now 1786333238880008 us; error 53 us; skew 500 ppm
Aug 10 03:40:38 dist-test-slave-ss4n krb5kdc[30853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333238, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:38.883580 31412 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:40:38.884766 31412 webserver.cc:460] Webserver started at http://127.21.175.126:44365/ using document root <none> and password file <none>
I20260810 03:40:38.885452 31412 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:38.885671 31412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:38.890210 31412 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:40:38.892781 31427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:38.894344 31412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:38.894519 31412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "41131c3d33f544f59d19c2c84fe8d08e"
format_stamp: "Formatted at 2026-08-10 03:40:37 on dist-test-slave-ss4n"
server_key: "3aa636320e16d59bbb26e87f91287a6e"
server_key_iv: "7868f42c09d4f64d562c3b95d3d0d5ca"
server_key_version: "encryptionkey@0"
I20260810 03:40:38.895141 31412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:38.939117 31412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:38.942915 31412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:38.943183 31412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:38.952121 31412 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:46473
I20260810 03:40:38.952136 31486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:46473 every 8 connection(s)
I20260810 03:40:38.953410 31412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:38.958053 31487 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:40:38.959295 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 31412
I20260810 03:40:38.959810 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:35555
--local_ip_for_outbound_sockets=127.21.175.65
--tserver_master_addrs=127.21.175.126:46473
--webserver_port=45821
--webserver_interface=127.21.175.65
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.175.65
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:40:38.960052 31487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e: Bootstrap starting.
I20260810 03:40:38.978389 31487 log.cc:824] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:38.985458 31487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:40:38.985936 31487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e: Bootstrap complete.
I20260810 03:40:38.990468 31487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41131c3d33f544f59d19c2c84fe8d08e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46473 } }
I20260810 03:40:38.990908 31487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41131c3d33f544f59d19c2c84fe8d08e, State: Initialized, Role: FOLLOWER
I20260810 03:40:38.991432 31487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41131c3d33f544f59d19c2c84fe8d08e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46473 } }
I20260810 03:40:38.991561 31487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:38.991657 31487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:38.991762 31487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:40:38.993942 31487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41131c3d33f544f59d19c2c84fe8d08e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46473 } }
I20260810 03:40:38.994297 31487 leader_election.cc:302] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41131c3d33f544f59d19c2c84fe8d08e; no voters:
I20260810 03:40:38.994694 31487 leader_election.cc:288] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 03:40:38.994788 31491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:40:38.995028 31491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [term 2 LEADER]: Becoming Leader. State: Replica: 41131c3d33f544f59d19c2c84fe8d08e, State: Running, Role: LEADER
I20260810 03:40:38.995350 31491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41131c3d33f544f59d19c2c84fe8d08e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46473 } }
I20260810 03:40:38.996019 31487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:38.996762 31492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "41131c3d33f544f59d19c2c84fe8d08e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41131c3d33f544f59d19c2c84fe8d08e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46473 } } }
I20260810 03:40:38.996902 31492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:38.997886 31493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41131c3d33f544f59d19c2c84fe8d08e. Latest consensus state: current_term: 2 leader_uuid: "41131c3d33f544f59d19c2c84fe8d08e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41131c3d33f544f59d19c2c84fe8d08e" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46473 } } }
I20260810 03:40:38.997987 31493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:38.997997 31498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:39.001371 31498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:39.002071 31498 catalog_manager.cc:1259] Loaded cluster ID: d1e30ecf25a94aa884b285eabd06a868
I20260810 03:40:39.002138 31498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:39.004558 31498 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:39.005220 31498 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 41131c3d33f544f59d19c2c84fe8d08e: Loaded TSK: 0
I20260810 03:40:39.006541 31498 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:40:39.086095 31489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:39.086398 31489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:39.086493 31489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:39.090750 31489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:39.090848 31489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:39.090947 31489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:40:39.096629 31489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.65.keytab
--principal=oryx/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:35555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=45821
--tserver_master_addrs=127.21.175.126:46473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:39.098136 31489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:39.099366 31489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:39.110565 31515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:39.110610 31516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:39.110594 31518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:39.111253 31489 server_base.cc:1034] running on GCE node
I20260810 03:40:39.111709 31489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:39.112440 31489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:39.113619 31489 hybrid_clock.cc:648] HybridClock initialized: now 1786333239113605 us; error 37 us; skew 500 ppm
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333239, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:39.116791 31489 init.cc:377] Logged in from keytab as oryx/127.21.175.65@KRBTEST.COM (short username oryx)
I20260810 03:40:39.118048 31489 webserver.cc:460] Webserver started at http://127.21.175.65:45821/ using document root <none> and password file <none>
I20260810 03:40:39.118673 31489 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:39.118794 31489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:39.123000 31489 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:40:39.125223 31525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:39.126667 31489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:39.126791 31489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "4efa09cf84a045468a88d118ea07a5cb"
format_stamp: "Formatted at 2026-08-10 03:40:38 on dist-test-slave-ss4n"
server_key: "853b6f8048a55ff3cd924ddf269f5f1c"
server_key_iv: "381bc83a9bd0d98cedc271a5999c4e63"
server_key_version: "encryptionkey@0"
I20260810 03:40:39.127418 31489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:39.147290 31489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:39.150473 31489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:39.150686 31489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:39.151335 31489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:39.152202 31489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:39.152248 31489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:39.152316 31489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:39.152348 31489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:39.163178 31489 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:35555
I20260810 03:40:39.163203 31648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:35555 every 8 connection(s)
I20260810 03:40:39.164259 31489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:40:39.168274 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 31489
I20260810 03:40:39.168846 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:38193
--local_ip_for_outbound_sockets=127.21.175.66
--tserver_master_addrs=127.21.175.126:46473
--webserver_port=44317
--webserver_interface=127.21.175.66
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.175.66
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 03:40:39.179337 31651 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:39.166919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:49521 (local address 127.21.175.126:46473)
0810 03:40:39.168334 (+ 1415us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:39.168358 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:39.168415 (+ 57us) server_negotiation.cc:407] Connection header received
0810 03:40:39.169386 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:39.169400 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:39.169780 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:39.170153 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:39.171427 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.172446 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.173744 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.174255 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.174453 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.178597 (+ 4144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:49521: BlockingRecv error: recv got EOF from 127.21.175.65:49521 (error 108)
Metrics: {"server-negotiator.queue_time_us":1250,"thread_start_us":143,"threads_started":1}
I20260810 03:40:39.179347 31652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:39.166931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:49521)
0810 03:40:39.167779 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:39.167841 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:39.168921 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:39.170224 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:39.170233 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:39.170719 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:39.171222 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.171237 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.172605 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.172610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.173607 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.173618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.174451 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.174457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.174671 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.175411 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:39.175428 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:39.178308 (+ 2880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":149,"threads_started":1}
W20260810 03:40:39.179943 31649 heartbeater.cc:644] Failed to heartbeat to 127.21.175.126:46473 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.175.126:46473: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:39.186371 31652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:39.180469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:35033)
0810 03:40:39.180692 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:39.180709 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:39.180815 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:39.181075 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:39.181081 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:39.181272 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:39.181398 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.181405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.182213 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.182217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.183122 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.183132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.183618 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.183622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.183718 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.184206 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:39.184220 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:39.186218 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134}
W20260810 03:40:39.186471 31651 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:39.180553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:35033 (local address 127.21.175.126:46473)
0810 03:40:39.180676 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:39.180679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:39.180776 (+ 97us) server_negotiation.cc:407] Connection header received
0810 03:40:39.180913 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:39.180915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:39.180956 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:39.181023 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:39.181589 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.182096 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.183264 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.183505 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.183590 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.186346 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:35033: BlockingRecv error: recv got EOF from 127.21.175.65:35033 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:39.192129 31652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:39.187119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:37605)
0810 03:40:39.187243 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:39.187259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:39.187338 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:39.187578 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:39.187581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:39.187740 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:39.187811 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.187816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.188528 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.188541 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.189054 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.189061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.189548 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.189551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.189630 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.190147 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:39.190158 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:39.191981 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 03:40:39.192226 31651 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:39.187223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:37605 (local address 127.21.175.126:46473)
0810 03:40:39.187362 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:39.187368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:39.187384 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:40:39.187422 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:39.187425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:39.187472 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:39.187556 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:39.187923 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.188417 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.189174 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.189407 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.189559 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.192113 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:37605: BlockingRecv error: recv got EOF from 127.21.175.65:37605 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 03:40:39.192365 31649 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:40:39.294090 31653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:39.294402 31653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:39.294448 31653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:39.299012 31653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:39.299085 31653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:39.299207 31653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:40:39.304953 31653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.66.keytab
--principal=oryx/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:38193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=44317
--tserver_master_addrs=127.21.175.126:46473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:39.306252 31653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:39.307230 31653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:39.317541 31658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:39.317656 31659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:39.317541 31661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:39.317956 31653 server_base.cc:1034] running on GCE node
I20260810 03:40:39.318571 31653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:39.319326 31653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:39.320549 31653 hybrid_clock.cc:648] HybridClock initialized: now 1786333239320511 us; error 56 us; skew 500 ppm
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333239, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:39.323678 31653 init.cc:377] Logged in from keytab as oryx/127.21.175.66@KRBTEST.COM (short username oryx)
I20260810 03:40:39.324671 31653 webserver.cc:460] Webserver started at http://127.21.175.66:44317/ using document root <none> and password file <none>
I20260810 03:40:39.325237 31653 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:39.325312 31653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:39.329622 31653 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:39.332099 31668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:39.333572 31653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:39.333683 31653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "00deef55fa4c4caa90576316b7cf2682"
format_stamp: "Formatted at 2026-08-10 03:40:38 on dist-test-slave-ss4n"
server_key: "65165154f65d104e0680213c54fbd772"
server_key_iv: "fcb22c899bf7c65abd91a3556ad40d75"
server_key_version: "encryptionkey@0"
I20260810 03:40:39.334295 31653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:39.356982 31653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:39.360855 31653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:39.361063 31653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:39.361845 31653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:39.362771 31653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:39.362820 31653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:39.362862 31653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:39.362913 31653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:39.372450 31653 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:38193
I20260810 03:40:39.372581 31799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:38193 every 8 connection(s)
I20260810 03:40:39.373661 31653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:40:39.379881 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 31653
I20260810 03:40:39.380363 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:46551
--local_ip_for_outbound_sockets=127.21.175.67
--tserver_master_addrs=127.21.175.126:46473
--webserver_port=38109
--webserver_interface=127.21.175.67
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.175.67
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 03:40:39.387522 31651 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:39.375792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:48515 (local address 127.21.175.126:46473)
0810 03:40:39.375987 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:39.375993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:39.377208 (+ 1215us) server_negotiation.cc:407] Connection header received
0810 03:40:39.378125 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:39.378129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:39.378188 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:39.378278 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:39.379584 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.380327 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.382657 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.383146 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.383306 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.387408 (+ 4102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:48515: BlockingRecv error: recv got EOF from 127.21.175.66:48515 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:40:39.388654 31803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:39.376269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:48515)
0810 03:40:39.377029 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:39.377073 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:39.377887 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:39.378503 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:39.378511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:39.378887 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:39.379389 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.379402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.380492 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.380498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.382090 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.382106 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.383332 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.383335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.383493 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.384164 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:39.384183 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:39.387263 (+ 3080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":147,"threads_started":1}
W20260810 03:40:39.389281 31800 heartbeater.cc:644] Failed to heartbeat to 127.21.175.126:46473 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.175.126:46473: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:39.395534 31803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:39.389823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:55887)
0810 03:40:39.389929 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:39.389944 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:39.390053 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:39.390339 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:39.390342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:39.390532 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:39.390644 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.390650 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.391391 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.391396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.391947 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.391956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.392412 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.392416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.392509 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.393148 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:39.393159 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:39.395377 (+ 2218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 03:40:39.395645 31651 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:39.389955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:55887 (local address 127.21.175.126:46473)
0810 03:40:39.390109 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:39.390114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:39.390129 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:40:39.390180 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:39.390188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:39.390228 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:39.390319 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:39.390762 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.391280 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.392075 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.392300 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.392408 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.395516 (+ 3108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:55887: BlockingRecv error: recv got EOF from 127.21.175.66:55887 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:39.402294 31803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:39.396316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:58729)
0810 03:40:39.396439 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:39.396453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:39.396523 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:39.396825 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:39.396827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:39.396993 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:39.397071 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.397076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.398279 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.398282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.398930 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.398940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.399541 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.399544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.399600 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.400255 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:39.400269 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:39.402194 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 03:40:39.402568 31651 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:39.396408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:58729 (local address 127.21.175.126:46473)
0810 03:40:39.396534 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:39.396562 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:39.396585 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:40:39.396633 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:39.396637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:39.396687 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:39.396786 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:39.397207 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.398169 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.399046 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.399435 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.399547 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.402398 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:58729: BlockingRecv error: recv got EOF from 127.21.175.66:58729 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 03:40:39.402658 31800 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:40:39.506999 31804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:39.507361 31804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:39.507426 31804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:39.511583 31804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:39.511655 31804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:39.511716 31804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:40:39.517243 31804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/krb5kdc/oryx_127.21.175.67.keytab
--principal=oryx/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:46551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=38109
--tserver_master_addrs=127.21.175.126:46473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:39.518409 31804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:39.519400 31804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:39.529884 31809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:39.529870 31810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:39.529870 31812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:39.530148 31804 server_base.cc:1034] running on GCE node
I20260810 03:40:39.530668 31804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:39.531478 31804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:39.532672 31804 hybrid_clock.cc:648] HybridClock initialized: now 1786333239532631 us; error 58 us; skew 500 ppm
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333239, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:39.536088 31804 init.cc:377] Logged in from keytab as oryx/127.21.175.67@KRBTEST.COM (short username oryx)
I20260810 03:40:39.537146 31804 webserver.cc:460] Webserver started at http://127.21.175.67:38109/ using document root <none> and password file <none>
I20260810 03:40:39.537842 31804 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:39.537930 31804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:39.542613 31804 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:40:39.545077 31819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:39.546772 31804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:39.546949 31804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "0d5420a5ba3e4e0d884bde254fd48e1f"
format_stamp: "Formatted at 2026-08-10 03:40:38 on dist-test-slave-ss4n"
server_key: "26b5a1f1dcd8dc0debba49b1d4c19ee8"
server_key_iv: "e81cec4d3135536c5db8d1e1682cfaff"
server_key_version: "encryptionkey@0"
I20260810 03:40:39.547575 31804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:39.570752 31804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:39.574364 31804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:39.574606 31804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:39.575266 31804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:39.576133 31804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:39.576180 31804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:39.576251 31804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:39.576289 31804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:39.585915 31804 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:46551
I20260810 03:40:39.585933 31942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:46551 every 8 connection(s)
I20260810 03:40:39.587014 31804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:40:39.589756 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 31804
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 03:40:39.599890 31651 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:39.588900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:35473 (local address 127.21.175.126:46473)
0810 03:40:39.589044 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:39.589055 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:39.590324 (+ 1269us) server_negotiation.cc:407] Connection header received
0810 03:40:39.591476 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:39.591485 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:39.591517 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:39.591573 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:39.593006 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.593584 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.595172 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.595377 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.595455 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.599764 (+ 4309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.67:35473: BlockingRecv error: recv got EOF from 127.21.175.67:35473 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:40:39.601107 31947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:39.589312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:35473)
0810 03:40:39.590133 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:39.590189 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:39.591240 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:39.591789 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:39.591800 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:39.592308 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:39.592865 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.592878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.593716 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.593720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.595033 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.595048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.595550 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.595554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.595799 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.596517 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:39.596540 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:39.599657 (+ 3117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":192,"threads_started":1}
I20260810 03:40:39.601186 31949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:39.590930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.21.175.126:46473)
0810 03:40:39.591164 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:39.591167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:39.591179 (+ 12us) server_negotiation.cc:407] Connection header received
0810 03:40:39.591221 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:39.591224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:39.591265 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:39.591347 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:39.592399 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.592935 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.594021 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.594248 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.594325 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.596008 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:39.596014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:39.596030 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:39.596057 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:39.598379 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:39.598910 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:39.598916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:39.598924 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:39.599142 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:39.599506 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:39.599510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:39.599513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:39.599918 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:39.600159 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:39.600739 (+ 580us) server_negotiation.cc:299] Negotiation successful
0810 03:40:39.601004 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":89,"threads_started":1}
W20260810 03:40:39.601801 31945 heartbeater.cc:644] Failed to heartbeat to 127.21.175.126:46473 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.175.126:46473: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:39.608115 31947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:39.602325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:58131)
0810 03:40:39.602543 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:39.602561 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:39.602668 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:39.602871 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:39.602874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:39.603050 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:39.603211 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.603218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.604163 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.604168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.604808 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.604817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.605360 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.605364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.605541 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.605991 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:39.606004 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:39.608001 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 03:40:39.608309 31949 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:39.602414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:58131 (local address 127.21.175.126:46473)
0810 03:40:39.602512 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:39.602518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:39.602611 (+ 93us) server_negotiation.cc:407] Connection header received
0810 03:40:39.602753 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:39.602756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:39.602788 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:39.602848 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:39.603344 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.604041 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.604952 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.605232 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.605355 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.608167 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.67:58131: BlockingRecv error: recv got EOF from 127.21.175.67:58131 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:39 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:39.614118 31947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:39.608695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:47113)
0810 03:40:39.608956 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:39.608967 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:39.609034 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:39.609326 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:39.609329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:39.609609 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:39.609703 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.609710 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.610437 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.610442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.611073 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:39.611083 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.611510 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.611513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:39.611578 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.612125 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:39.612137 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:39.613957 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199}
W20260810 03:40:39.614212 31949 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:39.608813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:47113 (local address 127.21.175.126:46473)
0810 03:40:39.608990 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:39.608993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:39.609007 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:40:39.609126 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:39.609128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:39.609164 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:39.609233 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:39.609829 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.610320 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.611188 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:39.611388 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:39.611465 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:39.614082 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.67:47113: BlockingRecv error: recv got EOF from 127.21.175.67:47113 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260810 03:40:39.614439 31945 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 03:40:40 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:40 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:40.198764 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:40.193087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:54849)
0810 03:40:40.193402 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:40.193416 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:40.193529 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:40.193798 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:40.193800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:40.193986 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:40.194087 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:40.194093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:40.194834 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:40.194837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:40.195405 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:40.195413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:40.195887 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:40.195890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:40.195963 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:40.196467 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:40.196478 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:40.198595 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":75,"threads_started":1}
W20260810 03:40:40.198909 31951 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:40.193163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:54849 (local address 127.21.175.126:46473)
0810 03:40:40.193458 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:40.193462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:40.193516 (+ 54us) server_negotiation.cc:407] Connection header received
0810 03:40:40.193631 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:40.193633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:40.193679 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:40.193758 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:40.194187 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:40.194736 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:40.195513 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:40.195761 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:40.195901 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:40.198770 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:54849: BlockingRecv error: recv got EOF from 127.21.175.65:54849 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":120,"threads_started":1}
Aug 10 03:40:40 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:40 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:40.409766 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:40.403395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:35171)
0810 03:40:40.403736 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:40.403752 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:40.403872 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:40.404181 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:40.404183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:40.404386 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:40.404571 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:40.404578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:40.405365 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:40.405368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:40.406217 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:40.406229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:40.406986 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:40.406990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:40.407101 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:40.407809 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:40.407822 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:40.409653 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":130,"threads_started":1}
W20260810 03:40:40.409919 31951 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:40.403487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:35171 (local address 127.21.175.126:46473)
0810 03:40:40.403659 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:40.403663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:40.403836 (+ 173us) server_negotiation.cc:407] Connection header received
0810 03:40:40.403996 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:40.403998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:40.404045 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:40.404126 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:40.404705 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:40.405225 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:40.406366 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:40.406788 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:40.406932 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:40.409801 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:35171: BlockingRecv error: recv got EOF from 127.21.175.66:35171 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 03:40:40 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:40 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:40.621304 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:40.615240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:49795)
0810 03:40:40.615538 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:40.615561 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:40.615667 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:40.616034 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:40.616040 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:40.616227 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:40.616347 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:40.616352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:40.617132 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:40.617135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:40.617837 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:40.617847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:40.618325 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:40.618328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:40.618412 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:40.619178 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:40.619190 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:40.621183 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":86,"threads_started":1}
W20260810 03:40:40.621479 31951 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:40.615315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:49795 (local address 127.21.175.126:46473)
0810 03:40:40.615476 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:40.615481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:40.615654 (+ 173us) server_negotiation.cc:407] Connection header received
0810 03:40:40.615819 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:40.615822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:40.615888 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:40.615979 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:40.616458 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:40.617018 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:40.617957 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:40.618189 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:40.618270 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:40.621346 (+ 3076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.67:49795: BlockingRecv error: recv got EOF from 127.21.175.67:49795 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 03:40:41 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:41 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:41.205965 31954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:41.199653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:50489)
0810 03:40:41.199947 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:41.199962 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:41.200084 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:41.200334 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:41.200336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:41.200592 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:41.200697 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:41.200703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:41.201567 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:41.201570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:41.202224 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:41.202234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:41.202786 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:41.202789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:41.202868 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:41.203535 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:41.203548 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:41.205803 (+ 2255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
W20260810 03:40:41.206063 31955 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:41.199675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:50489 (local address 127.21.175.126:46473)
0810 03:40:41.199980 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:41.199983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:41.200054 (+ 71us) server_negotiation.cc:407] Connection header received
0810 03:40:41.200167 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:41.200170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:41.200215 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:41.200322 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:41.200810 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:41.201348 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:41.202357 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:41.202641 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:41.202729 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:41.205950 (+ 3221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:50489: BlockingRecv error: recv got EOF from 127.21.175.65:50489 (error 108)
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":91,"threads_started":1}
Aug 10 03:40:41 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:41 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:41.420857 31956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:41.410786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:46391)
0810 03:40:41.411157 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:41.411172 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:41.411323 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:41.411677 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:41.411680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:41.411917 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:41.412025 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:41.412031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:41.412891 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:41.412897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:41.413567 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:41.413577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:41.414113 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:41.414117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:41.414208 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:41.418284 (+ 4076us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:41.418301 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:41.420612 (+ 2311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":93,"threads_started":1}
W20260810 03:40:41.421042 31955 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:41.410874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:46391 (local address 127.21.175.126:46473)
0810 03:40:41.411089 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:41.411095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:41.411263 (+ 168us) server_negotiation.cc:407] Connection header received
0810 03:40:41.411480 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:41.411484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:41.411546 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:41.411648 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:41.412161 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:41.412768 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:41.413707 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:41.413980 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:41.414109 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:41.420901 (+ 6792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:46391: BlockingRecv error: recv got EOF from 127.21.175.66:46391 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 10 03:40:41 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:41 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:41.628605 31957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:41.622136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:42515)
0810 03:40:41.622428 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:41.622442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:41.622543 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:41.622846 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:41.622848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:41.623070 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:41.623204 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:41.623211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:41.624113 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:41.624116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:41.624770 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:41.624780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:41.625350 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:41.625353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:41.625427 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:41.626108 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:41.626121 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:41.628436 (+ 2315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":102,"threads_started":1}
W20260810 03:40:41.628742 31955 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:41.622326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:42515 (local address 127.21.175.126:46473)
0810 03:40:41.622458 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:41.622461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:41.622493 (+ 32us) server_negotiation.cc:407] Connection header received
0810 03:40:41.622630 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:41.622632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:41.622688 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:41.622766 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:41.623369 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:41.623932 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:41.624920 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:41.625196 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:41.625314 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:41.628611 (+ 3297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.67:42515: BlockingRecv error: recv got EOF from 127.21.175.67:42515 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 03:40:42 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:42 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:42.212877 31958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:42.206997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:36859)
0810 03:40:42.207306 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:42.207330 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:42.207436 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:42.207718 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:42.207720 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:42.207916 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:42.208037 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:42.208042 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:42.209065 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:42.209070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:42.209779 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:42.209791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:42.210277 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:42.210281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:42.210368 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:42.210886 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:42.210897 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:42.212759 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
W20260810 03:40:42.213102 31959 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:42.207142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:36859 (local address 127.21.175.126:46473)
0810 03:40:42.207400 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:42.207403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:42.207420 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:40:42.207532 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:42.207534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:42.207579 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:42.207694 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:42.208193 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:42.208945 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:42.209905 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:42.210155 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:42.210270 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:42.212938 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:36859: BlockingRecv error: recv got EOF from 127.21.175.65:36859 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
Aug 10 03:40:42 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:42 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:42.428189 31960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:42.421955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:39621)
0810 03:40:42.422268 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:42.422283 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:42.422406 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:42.422676 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:42.422679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:42.422881 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:42.423015 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:42.423022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:42.423782 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:42.423785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:42.424415 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:42.424425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:42.424984 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:42.424986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:42.425054 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:42.425807 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:42.425819 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:42.428001 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":107,"threads_started":1}
W20260810 03:40:42.428558 31959 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:42.422062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:39621 (local address 127.21.175.126:46473)
0810 03:40:42.422244 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:42.422247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:42.422373 (+ 126us) server_negotiation.cc:407] Connection header received
0810 03:40:42.422497 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:42.422499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:42.422544 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:42.422642 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:42.423132 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:42.423653 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:42.424581 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:42.424859 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:42.424962 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:42.428210 (+ 3248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:39621: BlockingRecv error: recv got EOF from 127.21.175.66:39621 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 10 03:40:42 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:42 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:42.636009 31961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:42.629702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:49849)
0810 03:40:42.629993 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:42.630015 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:42.630127 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:42.630504 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:42.630506 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:42.630744 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:42.630865 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:42.630871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:42.631893 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:42.631895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:42.632599 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:42.632609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:42.633117 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:42.633121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:42.633190 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:42.633811 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:42.633823 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:42.635818 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":103,"threads_started":1}
W20260810 03:40:42.636165 31959 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:42.629763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:49849 (local address 127.21.175.126:46473)
0810 03:40:42.629925 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:42.629929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:42.630117 (+ 188us) server_negotiation.cc:407] Connection header received
0810 03:40:42.630209 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:42.630212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:42.630323 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:42.630482 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:42.631011 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:42.631756 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:42.632726 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:42.632981 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:42.633078 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:42.636012 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.67:49849: BlockingRecv error: recv got EOF from 127.21.175.67:49849 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 03:40:43 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:43 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:43.220237 31962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:43.213849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:35251)
0810 03:40:43.214206 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:43.214221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:43.214318 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:43.214630 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:43.214633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:43.214842 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:43.214963 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:43.214969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:43.215797 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:43.215800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:43.216439 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:43.216449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:43.217035 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:43.217038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:43.217112 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:43.218076 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:43.218089 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:43.220106 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":105,"threads_started":1}
W20260810 03:40:43.220474 31964 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:43.213936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:35251 (local address 127.21.175.126:46473)
0810 03:40:43.214285 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:43.214289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:43.214307 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:40:43.214408 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:43.214411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:43.214461 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:43.214556 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:43.215128 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:43.215676 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:43.216673 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:43.216908 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:43.216990 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:43.220309 (+ 3319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:35251: BlockingRecv error: recv got EOF from 127.21.175.65:35251 (error 108)
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":160,"threads_started":1}
Aug 10 03:40:43 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:43 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:43.435642 31965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:43.429296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:55997)
0810 03:40:43.429698 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:43.429718 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:43.429865 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:43.430254 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:43.430261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:43.430477 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:43.430677 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:43.430686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:43.431561 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:43.431568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:43.432284 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:43.432294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:43.432820 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:43.432824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:43.432919 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:43.433467 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:43.433501 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:43.435470 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":145,"threads_started":1}
W20260810 03:40:43.435729 31964 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:43.429427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:55997 (local address 127.21.175.126:46473)
0810 03:40:43.429595 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:43.429641 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:43.429820 (+ 179us) server_negotiation.cc:407] Connection header received
0810 03:40:43.429988 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:43.429991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:43.430051 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:43.430166 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:43.430813 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:43.431425 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:43.432422 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:43.432681 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:43.432804 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:43.435634 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:55997: BlockingRecv error: recv got EOF from 127.21.175.66:55997 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 10 03:40:43 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:43 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:43.643528 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:43.637104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:59699)
0810 03:40:43.637389 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:43.637409 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:43.637607 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:43.637914 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:43.637919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:43.638151 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:43.638279 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:43.638290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:43.639343 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:43.639345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:43.640090 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:43.640101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:43.640597 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:43.640600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:43.640695 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:43.641294 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:43.641309 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:43.643404 (+ 2095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":102,"threads_started":1}
W20260810 03:40:43.643747 31964 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:43.637309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:59699 (local address 127.21.175.126:46473)
0810 03:40:43.637618 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:43.637622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:43.637675 (+ 53us) server_negotiation.cc:407] Connection header received
0810 03:40:43.637723 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:43.637726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:43.637775 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:43.637844 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:43.638423 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:43.639176 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:43.640223 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:43.640480 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:43.640588 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:43.643596 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.67:59699: BlockingRecv error: recv got EOF from 127.21.175.67:59699 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 10 03:40:44 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:44 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:44.227327 31967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:44.221178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:56037)
0810 03:40:44.221579 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:44.221595 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:44.221704 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:44.221959 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:44.221961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:44.222178 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:44.222278 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:44.222283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:44.223197 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:44.223199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:44.223765 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:44.223773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:44.224246 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:44.224249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:44.224312 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:44.224891 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:44.224903 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:44.227201 (+ 2298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":99,"threads_started":1}
W20260810 03:40:44.227499 31968 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:44.221329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:56037 (local address 127.21.175.126:46473)
0810 03:40:44.221599 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:44.221604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:44.221674 (+ 70us) server_negotiation.cc:407] Connection header received
0810 03:40:44.221789 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:44.221792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:44.221834 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:44.221912 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:44.222540 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:44.223084 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:44.223890 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:44.224135 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:44.224220 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:44.227360 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:56037: BlockingRecv error: recv got EOF from 127.21.175.65:56037 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
Aug 10 03:40:44 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:44 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:44.443547 31969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:44.436748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:53913)
0810 03:40:44.437110 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:44.437126 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:44.437297 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:44.437713 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:44.437718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:44.438016 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:44.438201 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:44.438211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:44.439191 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:44.439197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:44.439947 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:44.439961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:44.440511 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:44.440514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:44.440587 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:44.441154 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:44.441168 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:44.443413 (+ 2245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":155,"threads_started":1}
W20260810 03:40:44.443797 31968 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:44.436944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:53913 (local address 127.21.175.126:46473)
0810 03:40:44.437100 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:44.437104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:44.437238 (+ 134us) server_negotiation.cc:407] Connection header received
0810 03:40:44.437409 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:44.437412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:44.437459 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:44.437610 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:44.438329 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:44.439065 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:44.440116 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:44.440342 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:44.440436 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:44.443643 (+ 3207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:53913: BlockingRecv error: recv got EOF from 127.21.175.66:53913 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 03:40:44 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:44 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:44.651922 31970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:44.644785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:49837)
0810 03:40:44.645246 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:44.645260 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:44.645411 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:44.645797 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:44.645800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:44.646045 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:44.646150 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:44.646156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:44.647214 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:44.647218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:44.648184 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:44.648196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:44.648805 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:44.648808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:44.648879 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:44.649606 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:44.649621 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:44.651763 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":111,"threads_started":1}
W20260810 03:40:44.652348 31968 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:44.644954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:49837 (local address 127.21.175.126:46473)
0810 03:40:44.645172 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:44.645178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:44.645359 (+ 181us) server_negotiation.cc:407] Connection header received
0810 03:40:44.645597 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:44.645606 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:44.645662 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:44.645769 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:44.646304 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:44.647042 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:44.648327 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:44.648664 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:44.648759 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:44.652235 (+ 3476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.67:49837: BlockingRecv error: recv got EOF from 127.21.175.67:49837 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 10 03:40:45 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:45 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:45.235369 31971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:45.228354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:44931)
0810 03:40:45.228692 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:45.228707 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:45.228819 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:45.229139 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:45.229142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:45.229355 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:45.229597 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:45.229607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:45.230949 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:45.230953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:45.231572 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:45.231584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:45.232253 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:45.232257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:45.232356 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:45.232936 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:45.232950 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:45.235241 (+ 2291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":100,"threads_started":1}
W20260810 03:40:45.235594 31972 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:45.228578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:44931 (local address 127.21.175.126:46473)
0810 03:40:45.228884 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:45.228888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:45.228903 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:40:45.228954 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:45.228957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:45.229001 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:45.229089 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:45.229738 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:45.230561 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:45.231761 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:45.232124 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:45.232250 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:45.235395 (+ 3145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:44931: BlockingRecv error: recv got EOF from 127.21.175.65:44931 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":144,"threads_started":1}
Aug 10 03:40:45 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:45 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:45.451249 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:45.444456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:59677)
0810 03:40:45.444934 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:45.444953 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:45.445124 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:45.445551 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:45.445554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:45.445791 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:45.445890 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:45.445897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:45.446860 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:45.446863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:45.447649 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:45.447663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:45.448270 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:45.448274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:45.448373 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:45.449014 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:45.449028 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:45.451058 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":101,"threads_started":1}
W20260810 03:40:45.451336 31972 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:45.444675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:59677 (local address 127.21.175.126:46473)
0810 03:40:45.444936 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:45.444944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:45.445056 (+ 112us) server_negotiation.cc:407] Connection header received
0810 03:40:45.445271 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:45.445275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:45.445326 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:45.445526 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:45.446050 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:45.446707 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:45.447813 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:45.448125 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:45.448267 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:45.451208 (+ 2941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:59677: BlockingRecv error: recv got EOF from 127.21.175.66:59677 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
Aug 10 03:40:45 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:45 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:45.658434 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:45.652723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:40161)
0810 03:40:45.653036 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:45.653059 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:45.653163 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:45.653391 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:45.653394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:45.653689 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:45.653795 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:45.653801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:45.654638 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:45.654641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:45.655211 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:45.655220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:45.655700 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:45.655704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:45.655775 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:45.656291 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:45.656303 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:45.658294 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":89,"threads_started":1}
W20260810 03:40:45.658525 31972 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:45.652776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:40161 (local address 127.21.175.126:46473)
0810 03:40:45.652994 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:45.652998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:45.653128 (+ 130us) server_negotiation.cc:407] Connection header received
0810 03:40:45.653233 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:45.653236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:45.653280 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:45.653353 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:45.653926 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:45.654463 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:45.655326 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:45.655551 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:45.655639 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:45.658428 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.67:40161: BlockingRecv error: recv got EOF from 127.21.175.67:40161 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 10 03:40:46 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:46 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:46.242559 31975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:46.236222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:43645)
0810 03:40:46.236531 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:46.236544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:46.236636 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:46.236930 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:46.236934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:46.237158 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:46.237266 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:46.237272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:46.238257 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:46.238260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:46.238842 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:46.238850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:46.239470 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:46.239473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:46.239564 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:46.240197 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:46.240209 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:46.242370 (+ 2161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":96,"threads_started":1}
W20260810 03:40:46.242652 31976 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:46.236367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:43645 (local address 127.21.175.126:46473)
0810 03:40:46.236649 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:46.236653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:46.236669 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:40:46.236710 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:46.236713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:46.236756 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:46.236845 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:46.237394 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:46.238137 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:46.239045 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:46.239330 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:46.239431 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:46.242538 (+ 3107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:43645: BlockingRecv error: recv got EOF from 127.21.175.65:43645 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":132,"threads_started":1}
Aug 10 03:40:46 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:46 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:46.465855 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:46.452137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:33717)
0810 03:40:46.452537 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:46.452550 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:46.452713 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:46.452991 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:46.452994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:46.453217 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:46.453310 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:46.453316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:46.454136 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:46.454139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:46.454703 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:46.454711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:46.455191 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:46.455194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:46.455282 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:46.463315 (+ 8033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:46.463334 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:46.465715 (+ 2381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":115,"threads_started":1}
W20260810 03:40:46.466184 31976 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:46.452204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:33717 (local address 127.21.175.126:46473)
0810 03:40:46.452399 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:46.452405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:46.452657 (+ 252us) server_negotiation.cc:407] Connection header received
0810 03:40:46.452821 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:46.452824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:46.452877 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:46.452965 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:46.453430 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:46.454019 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:46.454820 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:46.455074 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:46.455186 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:46.465897 (+ 10711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:33717: BlockingRecv error: recv got EOF from 127.21.175.66:33717 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 10 03:40:46 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:46 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:46.665668 31978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:46.659421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:48479)
0810 03:40:46.659872 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:46.659885 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:46.660033 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:46.660347 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:46.660350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:46.660569 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:46.660663 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:46.660669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:46.661428 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:46.661433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:46.662178 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:46.662188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:46.662705 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:46.662711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:46.662852 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:46.663477 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:46.663490 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:46.665468 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":123,"threads_started":1}
W20260810 03:40:46.665766 31976 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:46.659618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:48479 (local address 127.21.175.126:46473)
0810 03:40:46.659806 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:46.659812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:46.659973 (+ 161us) server_negotiation.cc:407] Connection header received
0810 03:40:46.660169 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:46.660173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:46.660227 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:46.660317 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:46.660784 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:46.661315 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:46.662297 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:46.662522 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:46.662641 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:46.665642 (+ 3001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.67:48479: BlockingRecv error: recv got EOF from 127.21.175.67:48479 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 10 03:40:47 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:47 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:47.249684 31980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:47.243453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:37051)
0810 03:40:47.243849 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:47.243864 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:47.243974 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:47.244265 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:47.244268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:47.244474 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:47.244643 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:47.244650 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:47.245395 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:47.245398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:47.246160 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:47.246171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:47.246688 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:47.246698 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:47.246775 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:47.247301 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:47.247314 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:47.249455 (+ 2141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":171,"threads_started":1}
W20260810 03:40:47.249827 31981 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:47.243677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:37051 (local address 127.21.175.126:46473)
0810 03:40:47.243926 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:47.243930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:47.243944 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:40:47.244052 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:47.244054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:47.244100 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:47.244186 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:47.244767 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:47.245295 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:47.246296 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:47.246525 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:47.246656 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:47.249689 (+ 3033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:37051: BlockingRecv error: recv got EOF from 127.21.175.65:37051 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":67,"threads_started":1}
Aug 10 03:40:47 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:47 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:47.473169 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:47.466906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:44681)
0810 03:40:47.467284 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:47.467301 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:47.467517 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:47.467830 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:47.467836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:47.468058 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:47.468157 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:47.468162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:47.469121 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:47.469127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:47.469933 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:47.469943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:47.470392 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:47.470395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:47.470492 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:47.471094 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:47.471113 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:47.472996 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":102,"threads_started":1}
W20260810 03:40:47.473241 31981 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:47.467085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:44681 (local address 127.21.175.126:46473)
0810 03:40:47.467284 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:47.467290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:47.467397 (+ 107us) server_negotiation.cc:407] Connection header received
0810 03:40:47.467642 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:47.467648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:47.467705 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:47.467795 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:47.468283 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:47.468990 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:47.470046 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:47.470277 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:47.470386 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:47.473127 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:44681: BlockingRecv error: recv got EOF from 127.21.175.66:44681 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 10 03:40:47 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:47 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:47.672142 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:47.666539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:38349)
0810 03:40:47.666875 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:47.666891 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:47.667016 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:47.667338 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:47.667341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:47.667562 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:47.667667 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:47.667674 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:47.668452 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:47.668455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:47.669058 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:47.669067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:47.669522 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:47.669524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:47.669589 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:47.670092 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:47.670104 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:47.671963 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":122,"threads_started":1}
W20260810 03:40:47.672230 31981 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:47.666561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:38349 (local address 127.21.175.126:46473)
0810 03:40:47.666691 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:47.666694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:47.666975 (+ 281us) server_negotiation.cc:407] Connection header received
0810 03:40:47.667143 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:47.667146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:47.667195 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:47.667258 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:47.667788 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:47.668335 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:47.669167 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:47.669390 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:47.669466 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:47.672099 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.67:38349: BlockingRecv error: recv got EOF from 127.21.175.67:38349 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 03:40:48 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:48 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:48.256968 31985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:48.250539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:40063)
0810 03:40:48.250892 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:48.250926 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:48.251057 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:48.251374 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:48.251377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:48.251691 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:48.251816 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:48.251824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:48.252724 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:48.252728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:48.253387 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:48.253397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:48.253989 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:48.253993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:48.254071 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:48.254618 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:48.254630 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:48.256824 (+ 2194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":126,"threads_started":1}
W20260810 03:40:48.257309 31986 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:48.250763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:40063 (local address 127.21.175.126:46473)
0810 03:40:48.251010 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:48.251015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:48.251031 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:40:48.251173 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:48.251176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:48.251221 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:48.251300 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:48.251961 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:48.252576 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:48.253636 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:48.253866 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:48.253948 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:48.256983 (+ 3035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:40063: BlockingRecv error: recv got EOF from 127.21.175.65:40063 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
Aug 10 03:40:48 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:48 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:48.480576 31988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:48.474355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:43919)
0810 03:40:48.474652 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:48.474672 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:48.474846 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:48.475179 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:48.475182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:48.475430 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:48.475545 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:48.475551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:48.476410 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:48.476413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:48.477114 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:48.477125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:48.477641 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:48.477645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:48.477722 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:48.478271 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:48.478285 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:48.480391 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":98,"threads_started":1}
W20260810 03:40:48.480777 31986 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:48.474403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:43919 (local address 127.21.175.126:46473)
0810 03:40:48.474569 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:48.474574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:48.474869 (+ 295us) server_negotiation.cc:407] Connection header received
0810 03:40:48.474925 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:48.474928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:48.474984 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:48.475079 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:48.475746 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:48.476296 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:48.477231 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:48.477452 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:48.477596 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:48.480613 (+ 3017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:43919: BlockingRecv error: recv got EOF from 127.21.175.66:43919 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 03:40:48 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:48 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:48.679352 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:48.673159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:40689)
0810 03:40:48.673617 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:48.673632 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:48.673739 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:48.674030 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:48.674033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:48.674228 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:48.674332 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:48.674338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:48.675106 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:48.675109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:48.675716 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:48.675725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:48.676139 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:48.676142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:48.676229 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:48.676746 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:48.676762 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:48.679219 (+ 2457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":117,"threads_started":1}
W20260810 03:40:48.679472 31986 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:48.673238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:40689 (local address 127.21.175.126:46473)
0810 03:40:48.673388 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:48.673393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:48.673708 (+ 315us) server_negotiation.cc:407] Connection header received
0810 03:40:48.673819 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:48.673822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:48.673869 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:48.673938 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:48.674451 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:48.674980 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:48.675822 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:48.676033 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:48.676109 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:48.679355 (+ 3246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.67:40689: BlockingRecv error: recv got EOF from 127.21.175.67:40689 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 03:40:49 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:49 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:49.266089 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:49.257998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:59959)
0810 03:40:49.258357 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:49.258371 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:49.258467 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:49.258798 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:49.258804 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:49.259089 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:49.259266 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:49.259276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:49.260491 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:49.260497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:49.261682 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:49.261698 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:49.262318 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:49.262324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:49.262449 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:49.263244 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:49.263261 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:49.265877 (+ 2616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":114,"threads_started":1}
W20260810 03:40:49.266222 31991 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:49.258094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:59959 (local address 127.21.175.126:46473)
0810 03:40:49.258425 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:49.258432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:49.258452 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:40:49.258560 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:49.258565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:49.258632 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:49.258741 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:49.259392 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:49.260340 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:49.261852 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:49.262162 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:49.262297 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:49.266063 (+ 3766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:59959: BlockingRecv error: recv got EOF from 127.21.175.65:59959 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":95,"threads_started":1}
Aug 10 03:40:49 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:49 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:49.488317 31992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:49.481652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:37077)
0810 03:40:49.481943 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:49.481966 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:49.482076 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:49.482367 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:49.482369 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:49.482630 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:49.482740 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:49.482747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:49.483646 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:49.483650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:49.484271 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:49.484279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:49.485002 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:49.485011 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:49.485105 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:49.485887 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:49.485900 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:49.488162 (+ 2262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":104,"threads_started":1}
W20260810 03:40:49.488490 31991 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:49.481788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:37077 (local address 127.21.175.126:46473)
0810 03:40:49.481938 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:49.481942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:49.482044 (+ 102us) server_negotiation.cc:407] Connection header received
0810 03:40:49.482188 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:49.482191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:49.482239 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:49.482325 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:49.482879 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:49.483459 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:49.484443 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:49.484810 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:49.484924 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:49.488326 (+ 3402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:37077: BlockingRecv error: recv got EOF from 127.21.175.66:37077 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 03:40:49 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:49 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:49.686894 31994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:49.680212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:34381)
0810 03:40:49.680563 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:49.680582 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:49.680696 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:49.681049 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:49.681053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:49.681277 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:49.681375 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:49.681381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:49.682283 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:49.682290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:49.682915 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:49.682923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:49.683466 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:49.683470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:49.683565 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:49.684196 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:49.684209 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:49.686765 (+ 2556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
W20260810 03:40:49.687058 31991 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:49.680398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:34381 (local address 127.21.175.126:46473)
0810 03:40:49.680632 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:49.680638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:49.680655 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:40:49.680825 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:49.680830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:49.680889 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:49.680995 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:49.681545 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:49.682131 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:49.683078 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:49.683323 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:49.683447 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:49.686929 (+ 3482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.67:34381: BlockingRecv error: recv got EOF from 127.21.175.67:34381 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 03:40:50 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:50 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:50.273300 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:50.267184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:35295)
0810 03:40:50.267548 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:50.267563 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:50.267676 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:50.267980 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:50.267982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:50.268185 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:50.268291 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:50.268297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:50.269093 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:50.269096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:50.269840 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:50.269850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:50.270287 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:50.270289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:50.270354 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:50.271016 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:50.271031 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:50.273077 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
W20260810 03:40:50.273422 31996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:50.267388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:35295 (local address 127.21.175.126:46473)
0810 03:40:50.267639 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:50.267643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:50.267659 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:40:50.267771 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:50.267774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:50.267822 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:50.267902 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:50.268408 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:50.268983 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:50.269953 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:50.270182 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:50.270262 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:50.273268 (+ 3006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:35295: BlockingRecv error: recv got EOF from 127.21.175.65:35295 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
Aug 10 03:40:50 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:50 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:50.495826 31998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:50.489303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:41233)
0810 03:40:50.489631 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:50.489649 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:50.489765 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:50.490015 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:50.490017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:50.490222 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:50.490324 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:50.490330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:50.491269 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:50.491272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:50.491888 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:50.491897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:50.492491 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:50.492496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:50.492586 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:50.493372 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:50.493389 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:50.495677 (+ 2288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":117,"threads_started":1}
W20260810 03:40:50.496013 31996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:50.489441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:41233 (local address 127.21.175.126:46473)
0810 03:40:50.489648 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:50.489652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:50.489734 (+ 82us) server_negotiation.cc:407] Connection header received
0810 03:40:50.489854 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:50.489856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:50.489903 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:50.489967 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:50.490441 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:50.491127 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:50.492066 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:50.492315 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:50.492418 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:50.495862 (+ 3444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:41233: BlockingRecv error: recv got EOF from 127.21.175.66:41233 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Aug 10 03:40:50 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:50 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:50.694849 31999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:50.688041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:43509)
0810 03:40:50.688426 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:50.688445 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:50.688644 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:50.688960 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:50.688963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:50.689249 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:50.689382 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:50.689390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:50.690584 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:50.690588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:50.691339 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:50.691349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:50.691958 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:50.691961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:50.692035 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:50.692660 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:50.692672 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:50.694698 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":109,"threads_started":1}
W20260810 03:40:50.695050 31996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:50.688223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:43509 (local address 127.21.175.126:46473)
0810 03:40:50.688417 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:50.688423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:50.688547 (+ 124us) server_negotiation.cc:407] Connection header received
0810 03:40:50.688773 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:50.688777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:50.688836 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:50.688939 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:50.689552 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:50.690325 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:50.691503 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:50.691824 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:50.691947 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:50.694878 (+ 2931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.67:43509: BlockingRecv error: recv got EOF from 127.21.175.67:43509 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 03:40:51 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:51 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:51.280534 32000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:51.274338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:54793)
0810 03:40:51.274641 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:51.274657 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:51.274782 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:51.275150 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:51.275154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:51.275403 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:51.275506 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:51.275512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:51.276414 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:51.276417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:51.277078 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:51.277088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:51.277775 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:51.277777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:51.277841 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:51.278478 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:51.278490 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:51.280414 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":94,"threads_started":1}
W20260810 03:40:51.280776 32001 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:51.274455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:54793 (local address 127.21.175.126:46473)
0810 03:40:51.274719 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:51.274723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:51.274740 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:40:51.274874 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:51.274878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:51.274933 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:51.275047 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:51.275609 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:51.276271 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:51.277199 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:51.277424 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:51.277532 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:51.280625 (+ 3093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:54793: BlockingRecv error: recv got EOF from 127.21.175.65:54793 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
Aug 10 03:40:51 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:51 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:51.510082 32002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:51.496962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:44119)
0810 03:40:51.497321 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:51.497336 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:51.497483 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:51.497811 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:51.497814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:51.498054 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:51.498156 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:51.498162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:51.499122 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:51.499128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:51.499864 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:51.499881 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:51.500413 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:51.500418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:51.500512 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:51.507571 (+ 7059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:51.507588 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:51.509953 (+ 2365us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":104,"threads_started":1}
W20260810 03:40:51.510280 32001 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:51.497034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:44119 (local address 127.21.175.126:46473)
0810 03:40:51.497238 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:51.497244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:51.497427 (+ 183us) server_negotiation.cc:407] Connection header received
0810 03:40:51.497628 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:51.497634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:51.497687 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:51.497789 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:51.498299 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:51.498995 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:51.500006 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:51.500283 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:51.500409 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:51.510115 (+ 9706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:44119: BlockingRecv error: recv got EOF from 127.21.175.66:44119 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 10 03:40:51 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:51 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:51.702003 32003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:51.695842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:37641)
0810 03:40:51.696231 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:51.696250 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:51.696407 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:51.696725 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:51.696727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:51.696945 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:51.697039 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:51.697044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:51.697979 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:51.697983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:51.698555 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:51.698563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:51.699159 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:51.699163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:51.699249 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:51.699756 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:51.699767 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:51.701877 (+ 2110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":134,"threads_started":1}
W20260810 03:40:51.702306 32001 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:51.695953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:37641 (local address 127.21.175.126:46473)
0810 03:40:51.696152 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:51.696157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:51.696328 (+ 171us) server_negotiation.cc:407] Connection header received
0810 03:40:51.696498 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:51.696501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:51.696595 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:51.696698 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:51.697185 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:51.697865 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:51.698688 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:51.699028 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:51.699141 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:51.702091 (+ 2950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.67:37641: BlockingRecv error: recv got EOF from 127.21.175.67:37641 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 10 03:40:52 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:52 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:52.287539 32004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:52.281586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:36161)
0810 03:40:52.281932 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:52.281947 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:52.282062 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:52.282361 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:52.282363 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:52.282554 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:52.282652 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:52.282657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:52.283411 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:52.283414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:52.283989 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:52.283999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:52.284464 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:52.284469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:52.284544 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:52.285177 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:52.285189 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:52.287425 (+ 2236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":145,"threads_started":1}
W20260810 03:40:52.287794 32005 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:52.281666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:36161 (local address 127.21.175.126:46473)
0810 03:40:52.281933 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:52.281937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:52.282035 (+ 98us) server_negotiation.cc:407] Connection header received
0810 03:40:52.282173 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:52.282176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:52.282223 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:52.282312 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:52.282760 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:52.283284 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:52.284110 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:52.284328 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:52.284411 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:52.287611 (+ 3200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:36161: BlockingRecv error: recv got EOF from 127.21.175.65:36161 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":113,"threads_started":1}
Aug 10 03:40:52 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:52 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:52.516832 32006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:52.510971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:58841)
0810 03:40:52.511301 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:52.511315 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:52.511415 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:52.511737 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:52.511739 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:52.511925 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:52.512028 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:52.512033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:52.512896 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:52.512899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:52.513559 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:52.513570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:52.514025 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:52.514028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:52.514092 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:52.514704 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:52.514716 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:52.516652 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
W20260810 03:40:52.516897 32005 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:52.511143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:58841 (local address 127.21.175.126:46473)
0810 03:40:52.511321 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:52.511325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:52.511395 (+ 70us) server_negotiation.cc:407] Connection header received
0810 03:40:52.511544 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:52.511547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:52.511598 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:52.511680 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:52.512139 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:52.512770 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:52.513684 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:52.513925 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:52.514015 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:52.516788 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:58841: BlockingRecv error: recv got EOF from 127.21.175.66:58841 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 10 03:40:52 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:52 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:52.710043 32007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:52.703152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:50031)
0810 03:40:52.703445 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:52.703460 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:52.703571 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:52.703933 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:52.703940 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:52.704252 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:52.704435 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:52.704448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:52.705707 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:52.705710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:52.706362 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:52.706371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:52.706915 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:52.706918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:52.706994 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:52.707571 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:52.707584 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:52.709899 (+ 2315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":111,"threads_started":1}
W20260810 03:40:52.710179 32005 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:52.703313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:50031 (local address 127.21.175.126:46473)
0810 03:40:52.703462 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:52.703465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:52.703543 (+ 78us) server_negotiation.cc:407] Connection header received
0810 03:40:52.703666 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:52.703669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:52.703722 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:52.703825 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:52.704658 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:52.705558 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:52.706505 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:52.706772 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:52.706870 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:52.710042 (+ 3172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.67:50031: BlockingRecv error: recv got EOF from 127.21.175.67:50031 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 03:40:53 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:53 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:53.295186 32008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:53.288454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:39383)
0810 03:40:53.288743 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:53.288762 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:53.288889 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:53.289193 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:53.289195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:53.289395 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:53.289569 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:53.289582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:53.290408 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:53.290411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:53.291035 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:53.291046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:53.291697 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:53.291706 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:53.291806 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:53.292552 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:53.292567 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:53.295031 (+ 2464us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
W20260810 03:40:53.295439 32009 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:53.288590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:39383 (local address 127.21.175.126:46473)
0810 03:40:53.288952 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:53.288956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:53.288971 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:40:53.289011 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:53.289024 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:53.289068 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:53.289155 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:53.289705 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:53.290262 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:53.291191 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:53.291467 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:53.291571 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:53.295267 (+ 3696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:39383: BlockingRecv error: recv got EOF from 127.21.175.65:39383 (error 108)
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":86,"threads_started":1}
Aug 10 03:40:53 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:53 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:53.523463 32010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:53.517675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:42549)
0810 03:40:53.517972 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:53.517988 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:53.518159 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:53.518440 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:53.518443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:53.518694 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:53.518794 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:53.518800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:53.519576 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:53.519582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:53.520155 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:53.520163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:53.520645 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:53.520648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:53.520734 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:53.521311 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:53.521326 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:53.523340 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":95,"threads_started":1}
W20260810 03:40:53.523628 32009 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:53.517755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:42549 (local address 127.21.175.126:46473)
0810 03:40:53.517958 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:53.517963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:53.518094 (+ 131us) server_negotiation.cc:407] Connection header received
0810 03:40:53.518268 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:53.518271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:53.518324 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:53.518414 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:53.518920 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:53.519453 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:53.520284 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:53.520517 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:53.520633 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:53.523501 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:42549: BlockingRecv error: recv got EOF from 127.21.175.66:42549 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 10 03:40:53 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:53 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.67@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:53.716979 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:53.711223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.67:60277)
0810 03:40:53.711530 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:53.711544 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:53.711656 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:53.711908 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:53.711910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:53.712095 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:53.712218 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:53.712223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:53.713034 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:53.713036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:53.713627 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:53.713635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:53.714077 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:53.714080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:53.714154 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:53.714706 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:53.714718 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:53.716856 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":115,"threads_started":1}
W20260810 03:40:53.717206 32009 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:53.711300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:60277 (local address 127.21.175.126:46473)
0810 03:40:53.711473 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:53.711481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:53.711655 (+ 174us) server_negotiation.cc:407] Connection header received
0810 03:40:53.711734 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:53.711737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:53.711789 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:53.711865 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:53.712333 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:53.712933 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:53.713732 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:53.713971 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:53.714055 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:53.717052 (+ 2997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.67:60277: BlockingRecv error: recv got EOF from 127.21.175.67:60277 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 10 03:40:54 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:54 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.65@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:54.303448 32013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:54.296309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.65:44231)
0810 03:40:54.296646 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:54.296661 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:54.296762 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:54.297077 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:54.297081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:54.297362 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:54.297472 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:54.297540 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:54.298409 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:54.298416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:54.299386 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:54.299401 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:54.300161 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:54.300165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:54.300260 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:54.300876 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:54.300888 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:54.303303 (+ 2415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":96,"threads_started":1}
W20260810 03:40:54.303709 32014 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:54.296457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:44231 (local address 127.21.175.126:46473)
0810 03:40:54.296735 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:54.296738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:54.296755 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:40:54.296864 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:54.296867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:54.296914 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:54.297045 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:54.297679 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:54.298247 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:54.299569 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:54.300013 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:54.300172 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:54.303443 (+ 3271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.65:44231: BlockingRecv error: recv got EOF from 127.21.175.65:44231 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":70,"threads_started":1}
Aug 10 03:40:54 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:40:54 dist-test-slave-ss4n krb5kdc[30853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.175.66@KRBTEST.COM for oryx/127.21.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:40:54.530771 32015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:54.524723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46473 (local address 127.21.175.66:53275)
0810 03:40:54.525021 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:54.525034 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:54.525135 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:54.525399 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:54.525401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:54.525661 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:54.525764 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:54.525771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:54.526558 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:54.526562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:54.527155 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:54.527166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:54.527682 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:54.527684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:54.527746 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:54.528407 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:54.528420 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:54.530635 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.175.126:46473: Server oryx/127.21.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":95,"threads_started":1}
W20260810 03:40:54.530930 32014 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:54.524820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:53275 (local address 127.21.175.126:46473)
0810 03:40:54.525014 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:54.525018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:54.525160 (+ 142us) server_negotiation.cc:407] Connection header received
0810 03:40:54.525207 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:54.525209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:54.525257 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:54.525348 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:54.525880 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:54.526406 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:54.527287 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:54.527574 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:54.527661 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:54.530795 (+ 3134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.175.66:53275: BlockingRecv error: recv got EOF from 127.21.175.66:53275 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 03:40:54.599371 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 31489
I20260810 03:40:54.608217 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 31653
I20260810 03:40:54.615834 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 31804
I20260810 03:40:54.623896 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 31412
2026-08-10T03:40:54Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18321 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T03:40:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:54Z Disabled control of system clock
I20260810 03:40:54.659236 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:41843
--webserver_interface=127.21.175.126
--webserver_port=0
--builtin_ntp_servers=127.21.175.84:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:41843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 03:40:54.776919 32020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:54.777226 32020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:54.777262 32020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:54.781440 32020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:54.781553 32020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:54.781571 32020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:54.781589 32020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:54.781610 32020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:40:54.786713 32020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:41843
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:41843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:54.787910 32020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:54.789347 32020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:54.799301 32028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:54.799292 32030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:54.799436 32020 server_base.cc:1034] running on GCE node
W20260810 03:40:54.799301 32025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:54.800330 32020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:54.801769 32020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:54.803047 32020 hybrid_clock.cc:648] HybridClock initialized: now 1786333254803023 us; error 37 us; skew 500 ppm
I20260810 03:40:54.805413 32020 webserver.cc:460] Webserver started at http://127.21.175.126:39799/ using document root <none> and password file <none>
I20260810 03:40:54.806252 32020 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:54.806368 32020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:54.806646 32020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:54.809046 32020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "16e90a4dec95446090bfa45458c651bb"
format_stamp: "Formatted at 2026-08-10 03:40:54 on dist-test-slave-ss4n"
server_key: "62f0da92cd61646805f8368b35a79d30"
server_key_iv: "3f90fe3f221f3e17d3aefb260cd750ad"
server_key_version: "encryptionkey@0"
I20260810 03:40:54.809782 32020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "16e90a4dec95446090bfa45458c651bb"
format_stamp: "Formatted at 2026-08-10 03:40:54 on dist-test-slave-ss4n"
server_key: "62f0da92cd61646805f8368b35a79d30"
server_key_iv: "3f90fe3f221f3e17d3aefb260cd750ad"
server_key_version: "encryptionkey@0"
I20260810 03:40:54.814324 32020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:40:54.817325 32036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:54.818899 32020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:40:54.819105 32020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "16e90a4dec95446090bfa45458c651bb"
format_stamp: "Formatted at 2026-08-10 03:40:54 on dist-test-slave-ss4n"
server_key: "62f0da92cd61646805f8368b35a79d30"
server_key_iv: "3f90fe3f221f3e17d3aefb260cd750ad"
server_key_version: "encryptionkey@0"
I20260810 03:40:54.819262 32020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:54.847878 32020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:54.848734 32020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:54.848987 32020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:54.857206 32020 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:41843
I20260810 03:40:54.857254 32091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:41843 every 8 connection(s)
I20260810 03:40:54.858498 32020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:54.862304 32092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:54.866820 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 32020
I20260810 03:40:54.867070 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:40:54.867393 22205 external_mini_cluster.cc:1428] Setting key 48daf0b8e74b4e422fd21ca11f8db71a
I20260810 03:40:54.869978 32092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb: Bootstrap starting.
I20260810 03:40:54.873180 32092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:54.874266 32092 log.cc:824] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:54.876847 32092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb: No bootstrap required, opened a new log
I20260810 03:40:54.880214 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:54.869121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.21.175.126:41843)
0810 03:40:54.869957 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:54.869984 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:54.870037 (+ 53us) server_negotiation.cc:407] Connection header received
0810 03:40:54.870999 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:54.871025 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:54.871602 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:54.872037 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:54.872565 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:54.874272 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:54.875695 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:54.876413 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:54.876592 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:54.877721 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:54.877727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:54.877744 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:54.878272 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:54.878345 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:54.878373 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:54.878475 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:54.878647 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 03:40:54.878936 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":137,"threads_started":1}
I20260810 03:40:54.881095 32092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e90a4dec95446090bfa45458c651bb" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 41843 } }
I20260810 03:40:54.881376 32092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:54.881474 32092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16e90a4dec95446090bfa45458c651bb, State: Initialized, Role: FOLLOWER
I20260810 03:40:54.881992 32092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e90a4dec95446090bfa45458c651bb" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 41843 } }
I20260810 03:40:54.882177 32092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:54.882251 32092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:54.882422 32092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:54.883852 32092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e90a4dec95446090bfa45458c651bb" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 41843 } }
I20260810 03:40:54.884238 32092 leader_election.cc:302] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16e90a4dec95446090bfa45458c651bb; no voters:
I20260810 03:40:54.884603 32092 leader_election.cc:288] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:40:54.884811 32099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:54.885099 32099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [term 1 LEADER]: Becoming Leader. State: Replica: 16e90a4dec95446090bfa45458c651bb, State: Running, Role: LEADER
I20260810 03:40:54.885535 32099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e90a4dec95446090bfa45458c651bb" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 41843 } }
I20260810 03:40:54.886166 32092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:54.887646 32101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16e90a4dec95446090bfa45458c651bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e90a4dec95446090bfa45458c651bb" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 41843 } } }
I20260810 03:40:54.887811 32101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:54.888149 32102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16e90a4dec95446090bfa45458c651bb. Latest consensus state: current_term: 1 leader_uuid: "16e90a4dec95446090bfa45458c651bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e90a4dec95446090bfa45458c651bb" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 41843 } } }
I20260810 03:40:54.888244 32102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb [sys.catalog]: This master's current role is: LEADER
W20260810 03:40:54.892093 32116 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:40:54.892203 32116 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:40:54.892298 32109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:54.893134 32109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:54.898976 32109 catalog_manager.cc:1347] Generated new cluster ID: a94e428cbee9441a9e2328c99d11246d
I20260810 03:40:54.899065 32109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:54.907770 32109 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:40:54.908823 32109 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:54.919729 32109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16e90a4dec95446090bfa45458c651bb: Generated new TSK 0
I20260810 03:40:54.920578 32109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:40:54.929015 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:41843
--builtin_ntp_servers=127.21.175.84:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 03:40:55.058561 32120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:55.058897 32120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:55.058965 32120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:55.063151 32120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:55.063261 32120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:55.063356 32120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:40:55.068234 32120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:41843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:55.069449 32120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:55.070932 32120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:55.081575 32129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:55.081707 32126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:55.081733 32120 server_base.cc:1034] running on GCE node
W20260810 03:40:55.081779 32127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:55.082445 32120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:55.083272 32120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:55.084524 32120 hybrid_clock.cc:648] HybridClock initialized: now 1786333255084494 us; error 43 us; skew 500 ppm
I20260810 03:40:55.086758 32120 webserver.cc:460] Webserver started at http://127.21.175.65:42539/ using document root <none> and password file <none>
I20260810 03:40:55.087524 32120 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:55.087644 32120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:55.087891 32120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:55.090291 32120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "2cc85aa36e6f45c5b83422736c8fb247"
format_stamp: "Formatted at 2026-08-10 03:40:55 on dist-test-slave-ss4n"
server_key: "4e2d592919e79c91857d684f413f2db0"
server_key_iv: "2ab151bc16324e5c7e25c0ac4cb25e01"
server_key_version: "encryptionkey@0"
I20260810 03:40:55.090953 32120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "2cc85aa36e6f45c5b83422736c8fb247"
format_stamp: "Formatted at 2026-08-10 03:40:55 on dist-test-slave-ss4n"
server_key: "4e2d592919e79c91857d684f413f2db0"
server_key_iv: "2ab151bc16324e5c7e25c0ac4cb25e01"
server_key_version: "encryptionkey@0"
I20260810 03:40:55.095089 32120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:40:55.097913 32136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:55.099280 32120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:55.099431 32120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "2cc85aa36e6f45c5b83422736c8fb247"
format_stamp: "Formatted at 2026-08-10 03:40:55 on dist-test-slave-ss4n"
server_key: "4e2d592919e79c91857d684f413f2db0"
server_key_iv: "2ab151bc16324e5c7e25c0ac4cb25e01"
server_key_version: "encryptionkey@0"
I20260810 03:40:55.099565 32120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:55.137619 32120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:55.138449 32120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:55.138682 32120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:55.139344 32120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:55.140341 32120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:55.140419 32120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:55.140489 32120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:55.140525 32120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:55.151496 32120 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:37543
I20260810 03:40:55.151526 32257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:37543 every 8 connection(s)
I20260810 03:40:55.152544 32120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:40:55.158119 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 32120
I20260810 03:40:55.158219 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:40:55.158594 22205 external_mini_cluster.cc:1428] Setting key 6407730333cdb6bbaf5742656b15079a
I20260810 03:40:55.160895 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:41843
--builtin_ntp_servers=127.21.175.84:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 03:40:55.162744 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:55.154521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:52375 (local address 127.21.175.126:41843)
0810 03:40:55.154668 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:55.154671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:55.155817 (+ 1146us) server_negotiation.cc:407] Connection header received
0810 03:40:55.157343 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:55.157346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:55.157409 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:55.157546 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:55.158611 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:55.159318 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:55.160542 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:55.160784 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:55.160903 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:55.162224 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:55.162228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:55.162231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:55.162244 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:55.162266 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:55.162280 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:55.162364 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:55.162527 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 03:40:55.162593 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:40:55.163890 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:55.154927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:41843 (local address 127.21.175.65:52375)
0810 03:40:55.155646 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:55.155692 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:55.157077 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:55.157734 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:55.157741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:55.157847 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:55.158442 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:55.158459 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:55.159460 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:55.159468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:55.160403 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:55.160413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:55.160950 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:55.160954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:55.161209 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:55.161924 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:40:55.161926 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:55.161969 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:40:55.162038 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:55.162366 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:55.162379 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:55.162387 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 03:40:55.162529 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 03:40:55.162998 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":117,"threads_started":1}
I20260810 03:40:55.165256 32258 heartbeater.cc:347] Connected to a master server at 127.21.175.126:41843
I20260810 03:40:55.165552 32258 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:55.166100 32258 heartbeater.cc:511] Master 127.21.175.126:41843 requested a full tablet report, sending...
I20260810 03:40:55.167976 32054 ts_manager.cc:194] Registered new tserver with Master: 2cc85aa36e6f45c5b83422736c8fb247 (127.21.175.65:37543)
I20260810 03:40:55.170034 32054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.175.65:52375
W20260810 03:40:55.285681 32261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:55.285997 32261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:55.286041 32261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:55.290494 32261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:55.290571 32261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:55.290629 32261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:40:55.295580 32261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:41843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:55.296731 32261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:55.298199 32261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:55.308610 32270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:55.308656 32267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:55.308635 32261 server_base.cc:1034] running on GCE node
W20260810 03:40:55.308873 32268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:55.309324 32261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:55.310132 32261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:55.311365 32261 hybrid_clock.cc:648] HybridClock initialized: now 1786333255311348 us; error 28 us; skew 500 ppm
I20260810 03:40:55.313345 32261 webserver.cc:460] Webserver started at http://127.21.175.66:40621/ using document root <none> and password file <none>
I20260810 03:40:55.314026 32261 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:55.314131 32261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:55.314380 32261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:55.316501 32261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "4df362173f67450da3d1b8da66d6aca9"
format_stamp: "Formatted at 2026-08-10 03:40:55 on dist-test-slave-ss4n"
server_key: "5e8a2ec5d1aa4b648575c93c48b34d7f"
server_key_iv: "ec55bccc7955e90b7648bac44ef4a080"
server_key_version: "encryptionkey@0"
I20260810 03:40:55.317121 32261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "4df362173f67450da3d1b8da66d6aca9"
format_stamp: "Formatted at 2026-08-10 03:40:55 on dist-test-slave-ss4n"
server_key: "5e8a2ec5d1aa4b648575c93c48b34d7f"
server_key_iv: "ec55bccc7955e90b7648bac44ef4a080"
server_key_version: "encryptionkey@0"
I20260810 03:40:55.321221 32261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 03:40:55.324236 32279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:55.325783 32261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:55.325927 32261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "4df362173f67450da3d1b8da66d6aca9"
format_stamp: "Formatted at 2026-08-10 03:40:55 on dist-test-slave-ss4n"
server_key: "5e8a2ec5d1aa4b648575c93c48b34d7f"
server_key_iv: "ec55bccc7955e90b7648bac44ef4a080"
server_key_version: "encryptionkey@0"
I20260810 03:40:55.326057 32261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:55.347086 32261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:55.347909 32261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:55.348136 32261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:55.348778 32261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:55.349918 32261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:55.349995 32261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:55.350060 32261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:55.350112 32261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:55.359689 32261 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:35615
I20260810 03:40:55.359712 32403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:35615 every 8 connection(s)
I20260810 03:40:55.360759 32261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:40:55.369858 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 32261
I20260810 03:40:55.370004 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:40:55.370311 22205 external_mini_cluster.cc:1428] Setting key 74a004effb80614eaf5fe31662996755
I20260810 03:40:55.371271 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:55.362897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:37983 (local address 127.21.175.126:41843)
0810 03:40:55.363041 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:55.363045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:55.364154 (+ 1109us) server_negotiation.cc:407] Connection header received
0810 03:40:55.365673 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:55.365676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:55.365732 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:55.365820 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:55.366754 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:55.367355 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:55.368799 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:55.369056 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:55.369153 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:55.370738 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:55.370742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:55.370744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:55.370757 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:55.370779 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:55.370787 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:55.370877 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:55.371061 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 03:40:55.371129 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:40:55.372215 32406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:55.363320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:41843 (local address 127.21.175.66:37983)
0810 03:40:55.364017 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:55.364061 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:55.365372 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:55.365988 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:55.365995 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:55.366077 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:55.366567 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:55.366579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:55.367621 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:55.367625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:55.368660 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:55.368671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:55.369231 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:55.369234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:55.369383 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:55.370336 (+ 953us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:40:55.370338 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:55.370382 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:40:55.370502 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:55.370893 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:55.370903 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:55.370909 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 03:40:55.371060 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 03:40:55.371376 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":125,"threads_started":1}
I20260810 03:40:55.372660 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:41843
--builtin_ntp_servers=127.21.175.84:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 03:40:55.373621 32404 heartbeater.cc:347] Connected to a master server at 127.21.175.126:41843
I20260810 03:40:55.373912 32404 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:55.374616 32404 heartbeater.cc:511] Master 127.21.175.126:41843 requested a full tablet report, sending...
I20260810 03:40:55.375820 32054 ts_manager.cc:194] Registered new tserver with Master: 4df362173f67450da3d1b8da66d6aca9 (127.21.175.66:35615)
I20260810 03:40:55.376705 32054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.175.66:37983
W20260810 03:40:55.490911 32407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:55.491217 32407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:55.491259 32407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:55.495334 32407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:55.495400 32407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:55.495455 32407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:40:55.500213 32407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:41843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:55.501405 32407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:55.502839 32407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:55.513527 32413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:55.513679 32407 server_base.cc:1034] running on GCE node
W20260810 03:40:55.513597 32415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:55.513589 32412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:55.514272 32407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:55.514945 32407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:55.516156 32407 hybrid_clock.cc:648] HybridClock initialized: now 1786333255516142 us; error 35 us; skew 500 ppm
I20260810 03:40:55.518350 32407 webserver.cc:460] Webserver started at http://127.21.175.67:34873/ using document root <none> and password file <none>
I20260810 03:40:55.519006 32407 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:55.519110 32407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:55.519362 32407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:55.521631 32407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "7178ee1f5cea4b2b902680532b2f7154"
format_stamp: "Formatted at 2026-08-10 03:40:55 on dist-test-slave-ss4n"
server_key: "1f07b5e11dfa18e66247cf53e0eac347"
server_key_iv: "15b60854188c87ae71318a5e9a63c6c8"
server_key_version: "encryptionkey@0"
I20260810 03:40:55.522229 32407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "7178ee1f5cea4b2b902680532b2f7154"
format_stamp: "Formatted at 2026-08-10 03:40:55 on dist-test-slave-ss4n"
server_key: "1f07b5e11dfa18e66247cf53e0eac347"
server_key_iv: "15b60854188c87ae71318a5e9a63c6c8"
server_key_version: "encryptionkey@0"
I20260810 03:40:55.526306 32407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:40:55.529327 32422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:55.531165 32407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:40:55.531332 32407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "7178ee1f5cea4b2b902680532b2f7154"
format_stamp: "Formatted at 2026-08-10 03:40:55 on dist-test-slave-ss4n"
server_key: "1f07b5e11dfa18e66247cf53e0eac347"
server_key_iv: "15b60854188c87ae71318a5e9a63c6c8"
server_key_version: "encryptionkey@0"
I20260810 03:40:55.531483 32407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:55.550552 32407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:55.551349 32407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:55.551573 32407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:55.552201 32407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:55.553211 32407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:55.553260 32407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:55.553301 32407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:55.553362 32407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:55.562616 32407 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:46807
I20260810 03:40:55.562752 32547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:46807 every 8 connection(s)
I20260810 03:40:55.563619 32407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:40:55.571589 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 32407
I20260810 03:40:55.571718 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:40:55.572027 22205 external_mini_cluster.cc:1428] Setting key 352d9fcb37d032cc486de579cac0e96d
I20260810 03:40:55.574080 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:55.565746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:58619 (local address 127.21.175.126:41843)
0810 03:40:55.565947 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:55.565951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:55.567156 (+ 1205us) server_negotiation.cc:407] Connection header received
0810 03:40:55.568698 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:55.568701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:55.568759 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:55.568848 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:55.569875 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:55.570418 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:55.571641 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:55.571960 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:55.572089 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:55.573435 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:55.573438 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:55.573440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:55.573452 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:55.573472 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:55.573524 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:55.573612 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:55.573857 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 03:40:55.573927 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 03:40:55.575125 32551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:55.566223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:41843 (local address 127.21.175.67:58619)
0810 03:40:55.566973 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:55.567022 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:55.568460 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:55.569048 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:55.569055 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:55.569143 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:55.569718 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:55.569734 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:55.570554 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:55.570558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:55.571490 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:55.571500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:55.572125 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:55.572130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:55.572305 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:55.573043 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:40:55.573046 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:55.573125 (+ 79us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:40:55.573241 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:55.573615 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:55.573627 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:55.573634 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:40:55.573785 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 03:40:55.574123 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":124,"threads_started":1}
I20260810 03:40:55.576327 32548 heartbeater.cc:347] Connected to a master server at 127.21.175.126:41843
I20260810 03:40:55.576584 32548 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:55.577158 32548 heartbeater.cc:511] Master 127.21.175.126:41843 requested a full tablet report, sending...
I20260810 03:40:55.578253 32054 ts_manager.cc:194] Registered new tserver with Master: 7178ee1f5cea4b2b902680532b2f7154 (127.21.175.67:46807)
I20260810 03:40:55.578869 32054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.175.67:58619
I20260810 03:40:55.586899 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 03:40:55.590147 32095 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:55.589100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60828 (local address 127.21.175.126:41843)
0810 03:40:55.589282 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:55.589287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:55.589430 (+ 143us) server_negotiation.cc:407] Connection header received
0810 03:40:55.589606 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:55.589609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:55.589689 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:55.589782 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:55.590032 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60828: BlockingRecv error: recv got EOF from 127.0.0.1:60828 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 03:40:55.590732 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 32120
I20260810 03:40:55.599676 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 32261
I20260810 03:40:55.606838 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 32407
I20260810 03:40:55.613097 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 32020
2026-08-10T03:40:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (987 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T03:40:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:55Z Disabled control of system clock
I20260810 03:40:55.646714 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:42089
--webserver_interface=127.21.175.126
--webserver_port=0
--builtin_ntp_servers=127.21.175.84:45063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:42089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 03:40:55.765973 32561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:55.766278 32561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:55.766316 32561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:55.770555 32561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:55.770632 32561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:55.770648 32561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:55.770665 32561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:55.770681 32561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:40:55.775748 32561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:45063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:42089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:42089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:55.777009 32561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:55.778584 32561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:55.787546 32566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:55.787570 32570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:55.788529 32567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:55.788633 32561 server_base.cc:1034] running on GCE node
I20260810 03:40:55.789251 32561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:55.790544 32561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:55.791738 32561 hybrid_clock.cc:648] HybridClock initialized: now 1786333255791675 us; error 77 us; skew 500 ppm
I20260810 03:40:55.794111 32561 webserver.cc:460] Webserver started at http://127.21.175.126:41543/ using document root <none> and password file <none>
I20260810 03:40:55.794754 32561 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:55.794826 32561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:55.795123 32561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:55.797456 32561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "3b7ae6b7cd464de0bcfb9888c00b13ec"
format_stamp: "Formatted at 2026-08-10 03:40:55 on dist-test-slave-ss4n"
server_key: "1515bc8c6ad9c3ba4405c049f59370e2"
server_key_iv: "74f2174076014f4ae3b2c06873d07c84"
server_key_version: "encryptionkey@0"
I20260810 03:40:55.798193 32561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "3b7ae6b7cd464de0bcfb9888c00b13ec"
format_stamp: "Formatted at 2026-08-10 03:40:55 on dist-test-slave-ss4n"
server_key: "1515bc8c6ad9c3ba4405c049f59370e2"
server_key_iv: "74f2174076014f4ae3b2c06873d07c84"
server_key_version: "encryptionkey@0"
I20260810 03:40:55.802472 32561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:40:55.805639 32576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:55.807145 32561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:55.807312 32561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "3b7ae6b7cd464de0bcfb9888c00b13ec"
format_stamp: "Formatted at 2026-08-10 03:40:55 on dist-test-slave-ss4n"
server_key: "1515bc8c6ad9c3ba4405c049f59370e2"
server_key_iv: "74f2174076014f4ae3b2c06873d07c84"
server_key_version: "encryptionkey@0"
I20260810 03:40:55.807454 32561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:55.812879 32561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:55.813723 32561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:55.813935 32561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:55.821961 32561 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:42089
I20260810 03:40:55.821972 32635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:42089 every 8 connection(s)
I20260810 03:40:55.823063 32561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:55.823776 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 32561
I20260810 03:40:55.823916 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:40:55.824343 22205 external_mini_cluster.cc:1428] Setting key 3f3f96a640f3e9906e2fea63dfb95ac8
I20260810 03:40:55.826721 32636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:55.831987 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:55.826074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51622 (local address 127.21.175.126:42089)
0810 03:40:55.827524 (+ 1450us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:55.827546 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:55.827592 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:40:55.828502 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:55.828527 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:55.828997 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:55.829402 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:55.830454 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:55.830460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:55.830476 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:55.830489 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:55.830562 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:55.830586 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:55.830754 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:55.830825 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 03:40:55.831077 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1011,"thread_start_us":689,"threads_started":1}
I20260810 03:40:55.834081 32636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec: Bootstrap starting.
I20260810 03:40:55.836449 32636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:55.837284 32636 log.cc:824] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:55.839545 32636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec: No bootstrap required, opened a new log
I20260810 03:40:55.843336 32636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7ae6b7cd464de0bcfb9888c00b13ec" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42089 } }
I20260810 03:40:55.843575 32636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:55.843613 32636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7ae6b7cd464de0bcfb9888c00b13ec, State: Initialized, Role: FOLLOWER
I20260810 03:40:55.844080 32636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7ae6b7cd464de0bcfb9888c00b13ec" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42089 } }
I20260810 03:40:55.844206 32636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:55.844297 32636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:55.844398 32636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:55.845450 32636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7ae6b7cd464de0bcfb9888c00b13ec" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42089 } }
I20260810 03:40:55.845860 32636 leader_election.cc:302] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b7ae6b7cd464de0bcfb9888c00b13ec; no voters:
I20260810 03:40:55.846200 32636 leader_election.cc:288] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:40:55.846374 32641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:55.846778 32641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [term 1 LEADER]: Becoming Leader. State: Replica: 3b7ae6b7cd464de0bcfb9888c00b13ec, State: Running, Role: LEADER
I20260810 03:40:55.847234 32641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7ae6b7cd464de0bcfb9888c00b13ec" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42089 } }
I20260810 03:40:55.847653 32636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:55.849066 32642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b7ae6b7cd464de0bcfb9888c00b13ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7ae6b7cd464de0bcfb9888c00b13ec" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42089 } } }
I20260810 03:40:55.849248 32642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:55.849653 32644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b7ae6b7cd464de0bcfb9888c00b13ec. Latest consensus state: current_term: 1 leader_uuid: "3b7ae6b7cd464de0bcfb9888c00b13ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7ae6b7cd464de0bcfb9888c00b13ec" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 42089 } } }
I20260810 03:40:55.849768 32644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:55.849723 32656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:55.852573 32656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:55.858671 32656 catalog_manager.cc:1347] Generated new cluster ID: 304f8bc4bc6f45df88a93e2b17fefe49
I20260810 03:40:55.858757 32656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:55.879966 32656 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:40:55.880383 32656 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:55.891486 32656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b7ae6b7cd464de0bcfb9888c00b13ec: Generated new TSK 0
I20260810 03:40:55.892381 32656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:40:55.902912 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:42089
--builtin_ntp_servers=127.21.175.84:45063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 03:40:56.032397 32663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:56.032768 32663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:56.032845 32663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:56.037233 32663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:56.037338 32663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:56.037437 32663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:40:56.042640 32663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:45063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:42089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:56.043883 32663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:56.045375 32663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:56.056380 32670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:56.056391 32672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:56.056447 32663 server_base.cc:1034] running on GCE node
W20260810 03:40:56.056391 32668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:56.057124 32663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:56.057897 32663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:56.059098 32663 hybrid_clock.cc:648] HybridClock initialized: now 1786333256059081 us; error 37 us; skew 500 ppm
I20260810 03:40:56.061300 32663 webserver.cc:460] Webserver started at http://127.21.175.65:44581/ using document root <none> and password file <none>
I20260810 03:40:56.062036 32663 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:56.062143 32663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:56.062402 32663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:56.064532 32663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "7a43790e4cee4dfdb417d5383d474614"
format_stamp: "Formatted at 2026-08-10 03:40:56 on dist-test-slave-ss4n"
server_key: "8718b59902b00007548b0392736b48cd"
server_key_iv: "109c4094dc23780f20fd4d18189e736f"
server_key_version: "encryptionkey@0"
I20260810 03:40:56.065166 32663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "7a43790e4cee4dfdb417d5383d474614"
format_stamp: "Formatted at 2026-08-10 03:40:56 on dist-test-slave-ss4n"
server_key: "8718b59902b00007548b0392736b48cd"
server_key_iv: "109c4094dc23780f20fd4d18189e736f"
server_key_version: "encryptionkey@0"
I20260810 03:40:56.069154 32663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 03:40:56.072099 32679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:56.073416 32663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 03:40:56.073592 32663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "7a43790e4cee4dfdb417d5383d474614"
format_stamp: "Formatted at 2026-08-10 03:40:56 on dist-test-slave-ss4n"
server_key: "8718b59902b00007548b0392736b48cd"
server_key_iv: "109c4094dc23780f20fd4d18189e736f"
server_key_version: "encryptionkey@0"
I20260810 03:40:56.073733 32663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:56.078640 32663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:56.079300 32663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:56.079501 32663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:56.080070 32663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:56.081045 32663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:56.081111 32663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:56.081174 32663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:56.081226 32663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:56.090663 32663 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:35769
I20260810 03:40:56.090734 340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:35769 every 8 connection(s)
I20260810 03:40:56.091733 32663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:40:56.098070 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:56.093812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:46523 (local address 127.21.175.126:42089)
0810 03:40:56.094001 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:56.094005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:56.095053 (+ 1048us) server_negotiation.cc:407] Connection header received
0810 03:40:56.096359 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:56.096362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:56.096401 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:56.096481 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:56.097664 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:56.097667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:56.097670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:56.097681 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:56.097701 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:56.097708 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:56.097788 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:56.097916 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 03:40:56.097953 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 03:40:56.098996 343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:56.094201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:42089 (local address 127.21.175.65:46523)
0810 03:40:56.094881 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:56.094925 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:56.096159 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:56.096698 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:56.096705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:56.096794 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:56.097348 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:40:56.097350 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:56.097391 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:40:56.097460 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:56.097789 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:56.097796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:56.097800 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 03:40:56.097936 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 03:40:56.098184 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":116,"threads_started":1}
I20260810 03:40:56.100425 341 heartbeater.cc:347] Connected to a master server at 127.21.175.126:42089
I20260810 03:40:56.100680 341 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:56.101099 341 heartbeater.cc:511] Master 127.21.175.126:42089 requested a full tablet report, sending...
I20260810 03:40:56.101651 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 32663
I20260810 03:40:56.101778 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:40:56.102128 22205 external_mini_cluster.cc:1428] Setting key ad329fb3289a2a2d7ea129b8594162e7
I20260810 03:40:56.102907 32593 ts_manager.cc:194] Registered new tserver with Master: 7a43790e4cee4dfdb417d5383d474614 (127.21.175.65:35769)
I20260810 03:40:56.104749 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:42089
--builtin_ntp_servers=127.21.175.84:45063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 03:40:56.222319 344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:56.222612 344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:56.222687 344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:56.226707 344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:56.226766 344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:56.226863 344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:40:56.231698 344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:45063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:42089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:56.232911 344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:56.234351 344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:56.244175 350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:56.244284 349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:56.244400 344 server_base.cc:1034] running on GCE node
W20260810 03:40:56.244171 352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:56.244948 344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:56.245622 344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:56.246788 344 hybrid_clock.cc:648] HybridClock initialized: now 1786333256246762 us; error 34 us; skew 500 ppm
I20260810 03:40:56.248870 344 webserver.cc:460] Webserver started at http://127.21.175.66:39293/ using document root <none> and password file <none>
I20260810 03:40:56.249567 344 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:56.249668 344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:56.249892 344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:56.251979 344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "badb9fac8e0e4210b0f1d4ed8cda6cb1"
format_stamp: "Formatted at 2026-08-10 03:40:56 on dist-test-slave-ss4n"
server_key: "aea8bce658f021b3c68efe63ebe3bc80"
server_key_iv: "b2df5a16dd21f25fd6fd86b6b92529f1"
server_key_version: "encryptionkey@0"
I20260810 03:40:56.252792 344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "badb9fac8e0e4210b0f1d4ed8cda6cb1"
format_stamp: "Formatted at 2026-08-10 03:40:56 on dist-test-slave-ss4n"
server_key: "aea8bce658f021b3c68efe63ebe3bc80"
server_key_iv: "b2df5a16dd21f25fd6fd86b6b92529f1"
server_key_version: "encryptionkey@0"
I20260810 03:40:56.256935 344 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 03:40:56.259716 359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:56.260990 344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:40:56.261142 344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "badb9fac8e0e4210b0f1d4ed8cda6cb1"
format_stamp: "Formatted at 2026-08-10 03:40:56 on dist-test-slave-ss4n"
server_key: "aea8bce658f021b3c68efe63ebe3bc80"
server_key_iv: "b2df5a16dd21f25fd6fd86b6b92529f1"
server_key_version: "encryptionkey@0"
I20260810 03:40:56.261268 344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:56.266165 344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:56.266888 344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:56.267091 344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:56.267674 344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:56.268625 344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:56.268698 344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:56.268765 344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:56.268814 344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:56.278633 344 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:40859
I20260810 03:40:56.278661 484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:40859 every 8 connection(s)
I20260810 03:40:56.279665 344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:40:56.283483 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 344
I20260810 03:40:56.283612 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:40:56.283939 22205 external_mini_cluster.cc:1428] Setting key 848296cc72da0b99eca4d449c1c996aa
I20260810 03:40:56.286480 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:42089
--builtin_ntp_servers=127.21.175.84:45063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 03:40:56.287122 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:56.281701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:37495 (local address 127.21.175.126:42089)
0810 03:40:56.281851 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:56.281855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:56.283243 (+ 1388us) server_negotiation.cc:407] Connection header received
0810 03:40:56.285060 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:56.285064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:56.285098 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:56.285189 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:56.286544 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:56.286547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:56.286549 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:56.286587 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:56.286613 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:56.286628 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:56.286737 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:56.286937 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 03:40:56.286982 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:40:56.288007 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:56.282177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:42089 (local address 127.21.175.66:37495)
0810 03:40:56.283046 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:56.283103 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:56.284812 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:56.285381 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:56.285393 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:56.285536 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:56.286190 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:40:56.286194 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:56.286252 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:40:56.286348 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:56.286778 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:56.286786 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:56.286791 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 03:40:56.286972 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 03:40:56.287228 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260810 03:40:56.289187 485 heartbeater.cc:347] Connected to a master server at 127.21.175.126:42089
I20260810 03:40:56.289460 485 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:56.289923 485 heartbeater.cc:511] Master 127.21.175.126:42089 requested a full tablet report, sending...
I20260810 03:40:56.291057 32593 ts_manager.cc:194] Registered new tserver with Master: badb9fac8e0e4210b0f1d4ed8cda6cb1 (127.21.175.66:40859)
W20260810 03:40:56.405078 488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:56.405359 488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:56.405400 488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:56.409431 488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:56.409557 488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:56.409698 488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:40:56.414740 488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:45063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:42089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:56.415900 488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:56.417346 488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:56.427683 496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:56.427759 493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:56.427847 488 server_base.cc:1034] running on GCE node
W20260810 03:40:56.427688 494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:56.428448 488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:56.429239 488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:56.430415 488 hybrid_clock.cc:648] HybridClock initialized: now 1786333256430384 us; error 51 us; skew 500 ppm
I20260810 03:40:56.432605 488 webserver.cc:460] Webserver started at http://127.21.175.67:42719/ using document root <none> and password file <none>
I20260810 03:40:56.433257 488 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:56.433326 488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:56.433667 488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:56.435997 488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "21eabcc4fa4f4bad81654e0b0a557655"
format_stamp: "Formatted at 2026-08-10 03:40:56 on dist-test-slave-ss4n"
server_key: "cebe70814bc52a95f1d3897ffc594636"
server_key_iv: "973864db38c66d223ae2dd7557c21f4e"
server_key_version: "encryptionkey@0"
I20260810 03:40:56.436726 488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "21eabcc4fa4f4bad81654e0b0a557655"
format_stamp: "Formatted at 2026-08-10 03:40:56 on dist-test-slave-ss4n"
server_key: "cebe70814bc52a95f1d3897ffc594636"
server_key_iv: "973864db38c66d223ae2dd7557c21f4e"
server_key_version: "encryptionkey@0"
I20260810 03:40:56.440918 488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:40:56.443804 502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:56.445082 488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:56.445180 488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "21eabcc4fa4f4bad81654e0b0a557655"
format_stamp: "Formatted at 2026-08-10 03:40:56 on dist-test-slave-ss4n"
server_key: "cebe70814bc52a95f1d3897ffc594636"
server_key_iv: "973864db38c66d223ae2dd7557c21f4e"
server_key_version: "encryptionkey@0"
I20260810 03:40:56.445331 488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:56.450814 488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:56.451426 488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:56.451586 488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:56.452124 488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:56.453058 488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:56.453112 488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:56.453162 488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:56.453181 488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:56.463222 488 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:36041
I20260810 03:40:56.463279 625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:36041 every 8 connection(s)
I20260810 03:40:56.464354 488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:40:56.471082 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:56.466624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:49005 (local address 127.21.175.126:42089)
0810 03:40:56.466794 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:56.466798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:56.467738 (+ 940us) server_negotiation.cc:407] Connection header received
0810 03:40:56.469164 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:56.469167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:56.469215 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:56.469328 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:56.470570 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:56.470573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:56.470576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:40:56.470589 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:56.470615 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:40:56.470629 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:56.470734 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:56.470890 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 03:40:56.470940 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:40:56.471998 628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:56.466838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:42089 (local address 127.21.175.67:49005)
0810 03:40:56.467563 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:56.467606 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:56.468917 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:56.469535 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:56.469543 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:56.469635 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:56.470286 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:40:56.470288 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:56.470332 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:40:56.470404 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:56.470725 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:56.470734 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:56.470741 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:40:56.470948 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 03:40:56.471203 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":118,"threads_started":1}
I20260810 03:40:56.473408 626 heartbeater.cc:347] Connected to a master server at 127.21.175.126:42089
I20260810 03:40:56.473743 626 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:56.474210 626 heartbeater.cc:511] Master 127.21.175.126:42089 requested a full tablet report, sending...
I20260810 03:40:56.474541 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 488
I20260810 03:40:56.474630 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:40:56.475032 22205 external_mini_cluster.cc:1428] Setting key e4945aab61ef00bfdbf9a355d6736c1c
I20260810 03:40:56.475646 32593 ts_manager.cc:194] Registered new tserver with Master: 21eabcc4fa4f4bad81654e0b0a557655 (127.21.175.67:36041)
I20260810 03:40:56.477651 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 03:40:56.480645 32639 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:40:56.479792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.21.175.126:42089)
0810 03:40:56.479935 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:56.479938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:56.480128 (+ 190us) server_negotiation.cc:407] Connection header received
0810 03:40:56.480232 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:56.480234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:56.480266 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:56.480337 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:56.480490 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51638: BlockingRecv error: recv got EOF from 127.0.0.1:51638 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:40:56.480798 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 32663
I20260810 03:40:56.491262 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 344
I20260810 03:40:56.502827 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 488
I20260810 03:40:56.510934 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 32561
2026-08-10T03:40:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (897 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:40:56 dist-test-slave-ss4n krb5kdc[636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:40:56 dist-test-slave-ss4n krb5kdc[636](info): set up 2 sockets
Aug 10 03:40:56 dist-test-slave-ss4n krb5kdc[636](info): commencing operation
krb5kdc: starting...
W20260810 03:40:57.715585 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:40:57 dist-test-slave-ss4n krb5kdc[636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:40:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:40:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:40:57.861771 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:46407
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:46407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:57.978744 651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:57.979066 651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:57.979110 651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:57.983248 651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:40:57.983315 651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:57.983335 651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:57.983353 651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:40:57.983369 651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:40:57.988981 651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:46407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:46407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:57.990113 651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:57.991098 651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:57.999980 659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:58.000017 657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:58.000005 656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:58.000047 651 server_base.cc:1034] running on GCE node
I20260810 03:40:58.000866 651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:58.002128 651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:58.003361 651 hybrid_clock.cc:648] HybridClock initialized: now 1786333258003346 us; error 32 us; skew 500 ppm
Aug 10 03:40:58 dist-test-slave-ss4n krb5kdc[636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333258, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:58.006240 651 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:40:58.007112 651 webserver.cc:460] Webserver started at http://127.21.175.126:38459/ using document root <none> and password file <none>
I20260810 03:40:58.007645 651 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:58.007704 651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:58.007879 651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:58.010023 651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "3e4cb5a6610c4508ad98922c0808b257"
format_stamp: "Formatted at 2026-08-10 03:40:58 on dist-test-slave-ss4n"
server_key: "fac1c8bc5a52c8ceb4a18ee9f67d4f76"
server_key_iv: "2fdb47bd5c477ba0ff510a67d72a6fc1"
server_key_version: "encryptionkey@0"
I20260810 03:40:58.010633 651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "3e4cb5a6610c4508ad98922c0808b257"
format_stamp: "Formatted at 2026-08-10 03:40:58 on dist-test-slave-ss4n"
server_key: "fac1c8bc5a52c8ceb4a18ee9f67d4f76"
server_key_iv: "2fdb47bd5c477ba0ff510a67d72a6fc1"
server_key_version: "encryptionkey@0"
I20260810 03:40:58.014695 651 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:40:58.017290 666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:58.018735 651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:58.018971 651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "3e4cb5a6610c4508ad98922c0808b257"
format_stamp: "Formatted at 2026-08-10 03:40:58 on dist-test-slave-ss4n"
server_key: "fac1c8bc5a52c8ceb4a18ee9f67d4f76"
server_key_iv: "2fdb47bd5c477ba0ff510a67d72a6fc1"
server_key_version: "encryptionkey@0"
I20260810 03:40:58.019102 651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:58.036880 651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:58.040441 651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:58.040685 651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:58.049008 651 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:46407
I20260810 03:40:58.049077 725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:46407 every 8 connection(s)
I20260810 03:40:58.050165 651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:40:58.053145 726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:58.058847 726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257: Bootstrap starting.
I20260810 03:40:58.059708 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 651
I20260810 03:40:58.059839 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:40:58.060192 22205 external_mini_cluster.cc:1428] Setting key d0ebe2967078e2e49e8ba4c3dc57655c
I20260810 03:40:58.062011 726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:58.062796 726 log.cc:824] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:58.065137 726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257: No bootstrap required, opened a new log
Aug 10 03:40:58 dist-test-slave-ss4n krb5kdc[636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:58.070041 726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4cb5a6610c4508ad98922c0808b257" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46407 } }
I20260810 03:40:58.070283 726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:58.070371 726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e4cb5a6610c4508ad98922c0808b257, State: Initialized, Role: FOLLOWER
I20260810 03:40:58.070952 726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4cb5a6610c4508ad98922c0808b257" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46407 } }
I20260810 03:40:58.071134 726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:40:58.071208 726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:40:58.071329 726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:58.072710 726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4cb5a6610c4508ad98922c0808b257" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46407 } }
I20260810 03:40:58.073109 726 leader_election.cc:302] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e4cb5a6610c4508ad98922c0808b257; no voters:
I20260810 03:40:58.073562 726 leader_election.cc:288] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:40:58.073695 732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:58.073933 732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [term 1 LEADER]: Becoming Leader. State: Replica: 3e4cb5a6610c4508ad98922c0808b257, State: Running, Role: LEADER
I20260810 03:40:58.074553 732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4cb5a6610c4508ad98922c0808b257" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46407 } }
I20260810 03:40:58.074930 726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:40:58.076504 733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e4cb5a6610c4508ad98922c0808b257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4cb5a6610c4508ad98922c0808b257" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46407 } } }
I20260810 03:40:58.076639 733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:58.077024 734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e4cb5a6610c4508ad98922c0808b257. Latest consensus state: current_term: 1 leader_uuid: "3e4cb5a6610c4508ad98922c0808b257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4cb5a6610c4508ad98922c0808b257" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46407 } } }
I20260810 03:40:58.077198 734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257 [sys.catalog]: This master's current role is: LEADER
I20260810 03:40:58.077811 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:58.061431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.21.175.126:46407)
0810 03:40:58.061925 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:58.061941 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:58.061978 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:40:58.062717 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:58.062736 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:58.063144 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:58.063552 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:58.064492 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.065756 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.066983 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.067518 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.067659 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:58.071025 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:58.071032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:58.071051 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:58.071089 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:58.073409 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:58.074092 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:58.074099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:58.074106 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:58.074278 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:58.074650 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:58.074654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:58.074656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:58.075240 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:58.075468 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:58.076284 (+ 816us) server_negotiation.cc:299] Negotiation successful
0810 03:40:58.076647 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":150,"threads_started":1}
W20260810 03:40:58.080226 747 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:40:58.080390 747 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:40:58.080500 744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:40:58.081214 744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:40:58.086687 744 catalog_manager.cc:1347] Generated new cluster ID: a7225462f3ea4d16b000ca8136e3ca3c
I20260810 03:40:58.086764 744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:40:58.097889 744 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:40:58.098886 744 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:40:58.104246 744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257: Generated new TSK 0
I20260810 03:40:58.105158 744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:40:58.146121 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46407
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:58.305922 753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:58.306234 753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:58.306280 753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:58.310482 753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:58.310557 753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:58.310672 753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:40:58.316267 753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:58.317453 753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:58.318596 753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:40:58.329707 753 server_base.cc:1034] running on GCE node
W20260810 03:40:58.329654 761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:58.329661 758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:58.329668 759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:58.330446 753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:58.331169 753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:58.332351 753 hybrid_clock.cc:648] HybridClock initialized: now 1786333258332299 us; error 65 us; skew 500 ppm
Aug 10 03:40:58 dist-test-slave-ss4n krb5kdc[636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333258, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:58.335520 753 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:40:58.336515 753 webserver.cc:460] Webserver started at http://127.21.175.65:45695/ using document root <none> and password file <none>
I20260810 03:40:58.337126 753 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:58.337227 753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:58.337520 753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:58.339730 753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "9a8d908e49d141dd99db70146a0a848b"
format_stamp: "Formatted at 2026-08-10 03:40:58 on dist-test-slave-ss4n"
server_key: "81a9c1c8a0aa2fa7c4ce02be116a741b"
server_key_iv: "44ef1877d1cbae7871cf8412d29e2507"
server_key_version: "encryptionkey@0"
I20260810 03:40:58.340376 753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "9a8d908e49d141dd99db70146a0a848b"
format_stamp: "Formatted at 2026-08-10 03:40:58 on dist-test-slave-ss4n"
server_key: "81a9c1c8a0aa2fa7c4ce02be116a741b"
server_key_iv: "44ef1877d1cbae7871cf8412d29e2507"
server_key_version: "encryptionkey@0"
I20260810 03:40:58.344383 753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:40:58.347151 770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:58.348407 753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 03:40:58.348553 753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "9a8d908e49d141dd99db70146a0a848b"
format_stamp: "Formatted at 2026-08-10 03:40:58 on dist-test-slave-ss4n"
server_key: "81a9c1c8a0aa2fa7c4ce02be116a741b"
server_key_iv: "44ef1877d1cbae7871cf8412d29e2507"
server_key_version: "encryptionkey@0"
I20260810 03:40:58.348726 753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:58.363488 753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:58.367182 753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:58.367413 753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:58.368073 753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:58.368950 753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:58.369026 753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:58.369093 753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:58.369143 753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:58.378916 753 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:43795
I20260810 03:40:58.378948 894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:43795 every 8 connection(s)
I20260810 03:40:58.379880 753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:40:58.385680 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 753
I20260810 03:40:58.385797 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:40:58.386147 22205 external_mini_cluster.cc:1428] Setting key ab83ebe28a80058deee428943b405e31
Aug 10 03:40:58 dist-test-slave-ss4n krb5kdc[636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333258, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
I20260810 03:40:58.396147 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:58.382057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:38407 (local address 127.21.175.126:46407)
0810 03:40:58.382214 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:58.382217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:58.383086 (+ 869us) server_negotiation.cc:407] Connection header received
0810 03:40:58.384615 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:58.384619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:58.384674 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:58.384779 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:58.386316 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.387127 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.388534 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.388830 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.388958 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:58.392001 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:58.392005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:58.392008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:58.392039 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:58.393819 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:58.394427 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:58.394431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:58.394439 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:58.394521 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:58.394924 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:58.394929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:58.394931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:58.395141 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:58.395266 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:58.395824 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 03:40:58.395966 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:40:58.397091 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:58.382304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46407 (local address 127.21.175.65:38407)
0810 03:40:58.382896 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:58.382948 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:58.384343 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:58.384943 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:58.384950 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:58.385354 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:58.386106 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:58.386124 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.387278 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.387282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:58.388382 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:58.388393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.389021 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.389025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:58.389216 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:58.389948 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:58.389966 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:58.391799 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:58.393961 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:58.393968 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:58.393987 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:58.394294 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:58.394644 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:58.394648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:58.394650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:58.394811 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:58.395266 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:58.395275 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:58.395556 (+ 281us) client_negotiation.cc:769] Sending connection context
0810 03:40:58.395779 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 03:40:58.396169 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":130,"threads_started":1}
I20260810 03:40:58.398536 896 heartbeater.cc:347] Connected to a master server at 127.21.175.126:46407
I20260810 03:40:58.398823 896 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:58.399425 896 heartbeater.cc:511] Master 127.21.175.126:46407 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
I20260810 03:40:58.401264 686 ts_manager.cc:194] Registered new tserver with Master: 9a8d908e49d141dd99db70146a0a848b (127.21.175.65:43795)
I20260810 03:40:58.402666 686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:38407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:40:58.420964 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46407
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:58.538805 901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:58.539103 901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:58.539151 901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:58.543331 901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:58.543409 901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:58.543519 901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:40:58.549118 901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:58.550310 901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:58.551452 901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:58.561805 906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:58.561898 901 server_base.cc:1034] running on GCE node
W20260810 03:40:58.561785 910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:58.561785 907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:58.562386 901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:58.562976 901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:58.564165 901 hybrid_clock.cc:648] HybridClock initialized: now 1786333258564127 us; error 48 us; skew 500 ppm
Aug 10 03:40:58 dist-test-slave-ss4n krb5kdc[636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333258, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:58.567183 901 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:40:58.568099 901 webserver.cc:460] Webserver started at http://127.21.175.66:40643/ using document root <none> and password file <none>
I20260810 03:40:58.568650 901 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:58.568712 901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:58.568879 901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:58.571027 901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "eaf7c6538fe446f6b5672b280e8531ee"
format_stamp: "Formatted at 2026-08-10 03:40:58 on dist-test-slave-ss4n"
server_key: "fbd8fc8bf0c6b8b16dc7bbedb5645690"
server_key_iv: "aeb3d9ccd9b870190c621fc7c01e9682"
server_key_version: "encryptionkey@0"
I20260810 03:40:58.571565 901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "eaf7c6538fe446f6b5672b280e8531ee"
format_stamp: "Formatted at 2026-08-10 03:40:58 on dist-test-slave-ss4n"
server_key: "fbd8fc8bf0c6b8b16dc7bbedb5645690"
server_key_iv: "aeb3d9ccd9b870190c621fc7c01e9682"
server_key_version: "encryptionkey@0"
I20260810 03:40:58.575511 901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 03:40:58.578464 917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:58.579936 901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:40:58.580057 901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "eaf7c6538fe446f6b5672b280e8531ee"
format_stamp: "Formatted at 2026-08-10 03:40:58 on dist-test-slave-ss4n"
server_key: "fbd8fc8bf0c6b8b16dc7bbedb5645690"
server_key_iv: "aeb3d9ccd9b870190c621fc7c01e9682"
server_key_version: "encryptionkey@0"
I20260810 03:40:58.580303 901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:58.595942 901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:58.599980 901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:58.600220 901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:58.600905 901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:58.601908 901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:58.601970 901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:58.602048 901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:58.602089 901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:58.612614 901 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:46609
I20260810 03:40:58.612649 1046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:46609 every 8 connection(s)
I20260810 03:40:58.613787 901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:40:58.618927 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 901
I20260810 03:40:58.619040 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:40:58.619331 22205 external_mini_cluster.cc:1428] Setting key d1f2d6a1daec929b47ed91c79f4e7cba
Aug 10 03:40:58 dist-test-slave-ss4n krb5kdc[636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333258, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:58.630949 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:58.616028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:46335 (local address 127.21.175.126:46407)
0810 03:40:58.616228 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:58.616233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:58.617140 (+ 907us) server_negotiation.cc:407] Connection header received
0810 03:40:58.618622 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:58.618629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:58.618699 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:58.618809 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:58.620303 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.621234 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.623137 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.623551 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.623674 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:58.626571 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:58.626579 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:58.626584 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:58.626621 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:58.628465 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:58.629031 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:58.629035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:58.629037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:58.629137 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:58.629531 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:58.629536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:58.629539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:58.629816 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:58.629933 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:58.630604 (+ 671us) server_negotiation.cc:299] Negotiation successful
0810 03:40:58.630752 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
I20260810 03:40:58.631978 1049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:58.616463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46407 (local address 127.21.175.66:46335)
0810 03:40:58.616956 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:58.617002 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:58.618367 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:58.618970 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:58.618977 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:58.619475 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:58.620120 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:58.620147 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.621402 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.621405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:58.622978 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:58.622993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.623758 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.623761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:58.623913 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:58.624684 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:58.624702 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:58.626325 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:58.628602 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:58.628607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:58.628621 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:58.628896 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:58.629235 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:58.629238 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:58.629239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:58.629381 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:58.629975 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:58.629982 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:58.630283 (+ 301us) client_negotiation.cc:769] Sending connection context
0810 03:40:58.630494 (+ 211us) client_negotiation.cc:240] Negotiation successful
0810 03:40:58.630933 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":126,"threads_started":1}
I20260810 03:40:58.633778 1047 heartbeater.cc:347] Connected to a master server at 127.21.175.126:46407
I20260810 03:40:58.634074 1047 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
I20260810 03:40:58.634671 1047 heartbeater.cc:511] Master 127.21.175.126:46407 requested a full tablet report, sending...
I20260810 03:40:58.636010 686 ts_manager.cc:194] Registered new tserver with Master: eaf7c6538fe446f6b5672b280e8531ee (127.21.175.66:46609)
I20260810 03:40:58.636969 686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:46335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:40:58.654603 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46407
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:40:58.773339 1052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:40:58.773712 1052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:40:58.773764 1052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:40:58.778138 1052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:40:58.778210 1052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:40:58.778272 1052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:40:58.783829 1052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:40:58.785055 1052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:40:58.786182 1052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:40:58.796658 1057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:58.796615 1058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:40:58.796645 1060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:40:58.796725 1052 server_base.cc:1034] running on GCE node
I20260810 03:40:58.797619 1052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:40:58.798343 1052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:40:58.799597 1052 hybrid_clock.cc:648] HybridClock initialized: now 1786333258799564 us; error 49 us; skew 500 ppm
Aug 10 03:40:58 dist-test-slave-ss4n krb5kdc[636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333258, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:40:58.802845 1052 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:40:58.803936 1052 webserver.cc:460] Webserver started at http://127.21.175.67:40631/ using document root <none> and password file <none>
I20260810 03:40:58.804651 1052 fs_manager.cc:359] Metadata directory not provided
I20260810 03:40:58.804767 1052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:40:58.805061 1052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:40:58.807521 1052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "b080dde8b76544179658d7fb61cde866"
format_stamp: "Formatted at 2026-08-10 03:40:58 on dist-test-slave-ss4n"
server_key: "1696cc4404c3ea3465b588dc385c86b9"
server_key_iv: "6379eed110dfb51dd9efed081d1a842d"
server_key_version: "encryptionkey@0"
I20260810 03:40:58.808236 1052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "b080dde8b76544179658d7fb61cde866"
format_stamp: "Formatted at 2026-08-10 03:40:58 on dist-test-slave-ss4n"
server_key: "1696cc4404c3ea3465b588dc385c86b9"
server_key_iv: "6379eed110dfb51dd9efed081d1a842d"
server_key_version: "encryptionkey@0"
I20260810 03:40:58.813172 1052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 03:40:58.816020 1068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:58.817443 1052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:40:58.817674 1052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "b080dde8b76544179658d7fb61cde866"
format_stamp: "Formatted at 2026-08-10 03:40:58 on dist-test-slave-ss4n"
server_key: "1696cc4404c3ea3465b588dc385c86b9"
server_key_iv: "6379eed110dfb51dd9efed081d1a842d"
server_key_version: "encryptionkey@0"
I20260810 03:40:58.817781 1052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:40:58.832948 1052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:40:58.836508 1052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:40:58.836683 1052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:40:58.837275 1052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:40:58.838234 1052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:40:58.838287 1052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:58.838328 1052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:40:58.838344 1052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:40:58.848033 1052 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:44667
I20260810 03:40:58.848057 1194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:44667 every 8 connection(s)
I20260810 03:40:58.849022 1052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:40:58.852938 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 1052
I20260810 03:40:58.853060 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:40:58.853366 22205 external_mini_cluster.cc:1428] Setting key 3cbce66e2ee9c01e4f9fa2f61276ac93
Aug 10 03:40:58 dist-test-slave-ss4n krb5kdc[636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333258, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:40:58.865123 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:58.851043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:57257 (local address 127.21.175.126:46407)
0810 03:40:58.851187 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:58.851191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:58.852225 (+ 1034us) server_negotiation.cc:407] Connection header received
0810 03:40:58.854261 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:58.854264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:58.854322 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:58.854416 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:58.856077 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.856616 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.857926 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.858154 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.858238 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:58.861179 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:58.861183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:58.861185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:58.861213 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:58.863026 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:58.863589 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:58.863592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:58.863593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:58.863675 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:58.863984 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:58.863986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:58.863987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:58.864172 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:58.864247 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:58.864815 (+ 568us) server_negotiation.cc:299] Negotiation successful
0810 03:40:58.864948 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:40:58.866114 1197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:58.851360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46407 (local address 127.21.175.67:57257)
0810 03:40:58.852044 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:58.852095 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:58.854003 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:58.854668 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:58.854676 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:58.855268 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:40:58.855927 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:58.855946 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.856737 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.856740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:58.857799 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:58.857810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.858315 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.858318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:58.858491 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:58.859259 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:40:58.859280 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:40:58.860969 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:40:58.863152 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:58.863158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:58.863173 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:58.863460 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:58.863759 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:40:58.863762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:40:58.863763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:40:58.863903 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:40:58.864260 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:40:58.864266 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:40:58.864525 (+ 259us) client_negotiation.cc:769] Sending connection context
0810 03:40:58.864768 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 03:40:58.865162 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":127,"threads_started":1}
I20260810 03:40:58.867383 1195 heartbeater.cc:347] Connected to a master server at 127.21.175.126:46407
I20260810 03:40:58.867686 1195 heartbeater.cc:464] Registering TS with master...
I20260810 03:40:58.868271 1195 heartbeater.cc:511] Master 127.21.175.126:46407 requested a full tablet report, sending...
I20260810 03:40:58.869416 685 ts_manager.cc:194] Registered new tserver with Master: b080dde8b76544179658d7fb61cde866 (127.21.175.67:44667)
I20260810 03:40:58.870074 685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:57257
I20260810 03:40:58.880498 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:40:58.891050 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:58.882719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37062 (local address 127.21.175.126:46407)
0810 03:40:58.882930 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:58.882935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:58.883085 (+ 150us) server_negotiation.cc:407] Connection header received
0810 03:40:58.883267 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:58.883271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:58.883321 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:58.883409 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:40:58.884225 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.884855 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.885849 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.886086 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.886211 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:58.887684 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:40:58.887688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:40:58.887691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:40:58.887726 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:40:58.889117 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:58.889627 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:58.889631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:58.889632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:58.889722 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:40:58.890013 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:40:58.890016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:40:58.890018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:40:58.890278 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:40:58.890422 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:58.890746 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 03:40:58.890866 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260810 03:40:58.894795 685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:40:58.897941 685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:40:58.913062 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:58.907000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:43795 (local address 127.0.0.1:40850)
0810 03:40:58.907451 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:58.907510 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:58.907649 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:58.908903 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:58.908913 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:58.908957 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:58.909147 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:58.909155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.910434 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.910449 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:58.911670 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:58.911679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.912505 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.912509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:58.912614 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:58.912650 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 03:40:58.912766 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 03:40:58.912892 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"spinlock_wait_cycles":1408,"thread_start_us":150,"threads_started":1}
I20260810 03:40:58.913062 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:58.907204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.21.175.65:43795)
0810 03:40:58.908250 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:58.908258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:58.908281 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:40:58.908373 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:58.908378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:58.908700 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:58.908889 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:58.909290 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.910309 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.911802 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.912364 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.912497 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:58.912660 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:58.912805 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 03:40:58.912902 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":73,"threads_started":1}
I20260810 03:40:58.914243 1209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:58.907342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:44667 (local address 127.0.0.1:35034)
0810 03:40:58.907899 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:58.907915 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:58.907996 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:58.909092 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:58.909095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:58.909115 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:58.909215 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:58.909222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.911080 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.911084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:58.912448 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:58.912460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.913830 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.913834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:58.913922 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:58.913946 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:40:58.914035 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 03:40:58.914105 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":101,"threads_started":1}
I20260810 03:40:58.914347 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:58.907991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35034 (local address 127.21.175.67:44667)
0810 03:40:58.908481 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:58.908491 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:58.908510 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:40:58.908576 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:58.908581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:58.908899 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:58.909112 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:58.909332 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.910889 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.912630 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.913667 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.913824 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:58.913953 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:58.914068 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 03:40:58.914180 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":178,"threads_started":1}
I20260810 03:40:58.915112 1210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:58.907908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:46609 (local address 127.0.0.1:59968)
0810 03:40:58.908844 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:58.908859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:58.908965 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:58.909611 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:58.909614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:58.909633 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:58.909721 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:58.909727 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.911200 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.911204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:58.912451 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:58.912468 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.913982 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.913989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:58.914094 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:58.914119 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:40:58.914871 (+ 752us) client_negotiation.cc:240] Negotiation successful
0810 03:40:58.914944 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"spinlock_wait_cycles":4736,"thread_start_us":110,"threads_started":1}
I20260810 03:40:58.915568 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:58.908382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.21.175.66:46609)
0810 03:40:58.908722 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:58.908738 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:58.909068 (+ 330us) server_negotiation.cc:407] Connection header received
0810 03:40:58.909154 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:58.909160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:58.909368 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:58.909580 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:58.909856 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.911051 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.912614 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:58.913580 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:58.913727 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:58.913907 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:58.915352 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0810 03:40:58.915448 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":143,"threads_started":1}
I20260810 03:40:58.916922 822 tablet_service.cc:1467] Processing CreateTablet for tablet ccc21c2d3b5c4a75b386bcf07e4e7424 (DEFAULT_TABLE table=test-table [id=7333432c1cf54c4fb476e55755b7d0dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:58.917429 1120 tablet_service.cc:1467] Processing CreateTablet for tablet ccc21c2d3b5c4a75b386bcf07e4e7424 (DEFAULT_TABLE table=test-table [id=7333432c1cf54c4fb476e55755b7d0dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:58.918298 822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc21c2d3b5c4a75b386bcf07e4e7424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:58.918030 968 tablet_service.cc:1467] Processing CreateTablet for tablet ccc21c2d3b5c4a75b386bcf07e4e7424 (DEFAULT_TABLE table=test-table [id=7333432c1cf54c4fb476e55755b7d0dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:40:58.918596 1120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc21c2d3b5c4a75b386bcf07e4e7424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:58.919075 968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc21c2d3b5c4a75b386bcf07e4e7424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:40:58.925559 1213 tablet_bootstrap.cc:492] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee: Bootstrap starting.
I20260810 03:40:58.927340 1215 tablet_bootstrap.cc:492] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866: Bootstrap starting.
I20260810 03:40:58.927340 1214 tablet_bootstrap.cc:492] T ccc21c2d3b5c4a75b386bcf07e4e7424 P 9a8d908e49d141dd99db70146a0a848b: Bootstrap starting.
I20260810 03:40:58.928391 1213 tablet_bootstrap.cc:654] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:58.929334 1213 log.cc:824] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:58.929800 1214 tablet_bootstrap.cc:654] T ccc21c2d3b5c4a75b386bcf07e4e7424 P 9a8d908e49d141dd99db70146a0a848b: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:58.930271 1215 tablet_bootstrap.cc:654] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866: Neither blocks nor log segments found. Creating new log.
I20260810 03:40:58.930838 1214 log.cc:824] T ccc21c2d3b5c4a75b386bcf07e4e7424 P 9a8d908e49d141dd99db70146a0a848b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:58.931160 1215 log.cc:824] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866: Log is configured to *not* fsync() on all Append() calls
I20260810 03:40:58.931470 1213 tablet_bootstrap.cc:492] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee: No bootstrap required, opened a new log
I20260810 03:40:58.931689 1213 ts_tablet_manager.cc:1397] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260810 03:40:58.932863 1214 tablet_bootstrap.cc:492] T ccc21c2d3b5c4a75b386bcf07e4e7424 P 9a8d908e49d141dd99db70146a0a848b: No bootstrap required, opened a new log
I20260810 03:40:58.933111 1214 ts_tablet_manager.cc:1397] T ccc21c2d3b5c4a75b386bcf07e4e7424 P 9a8d908e49d141dd99db70146a0a848b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:40:58.933238 1215 tablet_bootstrap.cc:492] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866: No bootstrap required, opened a new log
I20260810 03:40:58.933416 1215 ts_tablet_manager.cc:1397] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:40:58.935811 1213 raft_consensus.cc:348] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8d908e49d141dd99db70146a0a848b" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43795 } } peers { permanent_uuid: "eaf7c6538fe446f6b5672b280e8531ee" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 46609 } } peers { permanent_uuid: "b080dde8b76544179658d7fb61cde866" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 44667 } }
I20260810 03:40:58.936230 1213 raft_consensus.cc:374] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:58.936334 1213 raft_consensus.cc:729] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf7c6538fe446f6b5672b280e8531ee, State: Initialized, Role: FOLLOWER
I20260810 03:40:58.936882 1213 consensus_queue.cc:260] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8d908e49d141dd99db70146a0a848b" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43795 } } peers { permanent_uuid: "eaf7c6538fe446f6b5672b280e8531ee" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 46609 } } peers { permanent_uuid: "b080dde8b76544179658d7fb61cde866" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 44667 } }
I20260810 03:40:58.937289 1215 raft_consensus.cc:348] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8d908e49d141dd99db70146a0a848b" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43795 } } peers { permanent_uuid: "eaf7c6538fe446f6b5672b280e8531ee" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 46609 } } peers { permanent_uuid: "b080dde8b76544179658d7fb61cde866" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 44667 } }
I20260810 03:40:58.937439 1214 raft_consensus.cc:348] T ccc21c2d3b5c4a75b386bcf07e4e7424 P 9a8d908e49d141dd99db70146a0a848b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8d908e49d141dd99db70146a0a848b" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43795 } } peers { permanent_uuid: "eaf7c6538fe446f6b5672b280e8531ee" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 46609 } } peers { permanent_uuid: "b080dde8b76544179658d7fb61cde866" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 44667 } }
I20260810 03:40:58.937744 1215 raft_consensus.cc:374] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:58.937834 1215 raft_consensus.cc:729] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b080dde8b76544179658d7fb61cde866, State: Initialized, Role: FOLLOWER
I20260810 03:40:58.937837 1047 heartbeater.cc:503] Master 127.21.175.126:46407 was elected leader, sending a full tablet report...
I20260810 03:40:58.937855 1214 raft_consensus.cc:374] T ccc21c2d3b5c4a75b386bcf07e4e7424 P 9a8d908e49d141dd99db70146a0a848b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:40:58.937914 1214 raft_consensus.cc:729] T ccc21c2d3b5c4a75b386bcf07e4e7424 P 9a8d908e49d141dd99db70146a0a848b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a8d908e49d141dd99db70146a0a848b, State: Initialized, Role: FOLLOWER
I20260810 03:40:58.938213 1213 ts_tablet_manager.cc:1428] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260810 03:40:58.938402 1215 consensus_queue.cc:260] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8d908e49d141dd99db70146a0a848b" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43795 } } peers { permanent_uuid: "eaf7c6538fe446f6b5672b280e8531ee" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 46609 } } peers { permanent_uuid: "b080dde8b76544179658d7fb61cde866" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 44667 } }
I20260810 03:40:58.938423 1214 consensus_queue.cc:260] T ccc21c2d3b5c4a75b386bcf07e4e7424 P 9a8d908e49d141dd99db70146a0a848b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8d908e49d141dd99db70146a0a848b" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43795 } } peers { permanent_uuid: "eaf7c6538fe446f6b5672b280e8531ee" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 46609 } } peers { permanent_uuid: "b080dde8b76544179658d7fb61cde866" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 44667 } }
I20260810 03:40:58.939136 1195 heartbeater.cc:503] Master 127.21.175.126:46407 was elected leader, sending a full tablet report...
I20260810 03:40:58.939386 1215 ts_tablet_manager.cc:1428] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 03:40:58.939491 1214 ts_tablet_manager.cc:1428] T ccc21c2d3b5c4a75b386bcf07e4e7424 P 9a8d908e49d141dd99db70146a0a848b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 03:40:58.939837 896 heartbeater.cc:503] Master 127.21.175.126:46407 was elected leader, sending a full tablet report...
I20260810 03:40:59.068818 1220 raft_consensus.cc:482] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:40:59.069079 1220 raft_consensus.cc:504] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8d908e49d141dd99db70146a0a848b" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43795 } } peers { permanent_uuid: "eaf7c6538fe446f6b5672b280e8531ee" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 46609 } } peers { permanent_uuid: "b080dde8b76544179658d7fb61cde866" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 44667 } }
I20260810 03:40:59.070343 1220 leader_election.cc:288] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a8d908e49d141dd99db70146a0a848b (127.21.175.65:43795), b080dde8b76544179658d7fb61cde866 (127.21.175.67:44667)
I20260810 03:40:59.075814 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:59.070926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:39323 (local address 127.21.175.65:43795)
0810 03:40:59.071104 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:59.071109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:59.071127 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:40:59.071190 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:59.071192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:59.071235 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:59.071343 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:59.071748 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:59.072553 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:59.074351 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:59.075030 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:59.075165 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:59.075222 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:59.075605 (+ 383us) server_negotiation.cc:299] Negotiation successful
0810 03:40:59.075680 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260810 03:40:59.075814 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:59.070842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:49727 (local address 127.21.175.67:44667)
0810 03:40:59.070994 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:59.070998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:59.071326 (+ 328us) server_negotiation.cc:407] Connection header received
0810 03:40:59.071459 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:59.071467 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:59.071555 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:59.071620 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:40:59.071900 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:59.072621 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:59.074351 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:59.075031 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:59.075165 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:59.075222 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:59.075596 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 03:40:59.075680 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:40:59.075876 1223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:59.070688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:44667 (local address 127.21.175.66:49727)
0810 03:40:59.071214 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:59.071225 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:59.071343 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:59.071661 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:59.071667 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:59.071692 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:59.071790 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:59.071794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:59.072747 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:59.072750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:59.074212 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:59.074221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:59.075195 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:59.075199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:59.075350 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:59.075395 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 03:40:59.075503 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 03:40:59.075604 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":233,"thread_start_us":153,"threads_started":1}
I20260810 03:40:59.075830 1049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:59.070688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:43795 (local address 127.21.175.66:39323)
0810 03:40:59.070893 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:59.070912 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:40:59.071025 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:40:59.071368 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:40:59.071374 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:40:59.071394 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:40:59.071572 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:59.071579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:59.072691 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:59.072697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:59.074210 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:40:59.074221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:40:59.075195 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:40:59.075199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:40:59.075373 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:59.075392 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 03:40:59.075566 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 03:40:59.075619 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":151}
I20260810 03:40:59.076706 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc21c2d3b5c4a75b386bcf07e4e7424" candidate_uuid: "eaf7c6538fe446f6b5672b280e8531ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b080dde8b76544179658d7fb61cde866" is_pre_election: true
I20260810 03:40:59.076824 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc21c2d3b5c4a75b386bcf07e4e7424" candidate_uuid: "eaf7c6538fe446f6b5672b280e8531ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a8d908e49d141dd99db70146a0a848b" is_pre_election: true
I20260810 03:40:59.077080 1143 raft_consensus.cc:2454] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf7c6538fe446f6b5672b280e8531ee in term 0.
I20260810 03:40:59.077180 844 raft_consensus.cc:2454] T ccc21c2d3b5c4a75b386bcf07e4e7424 P 9a8d908e49d141dd99db70146a0a848b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf7c6538fe446f6b5672b280e8531ee in term 0.
I20260810 03:40:59.077724 918 leader_election.cc:302] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9a8d908e49d141dd99db70146a0a848b, b080dde8b76544179658d7fb61cde866, eaf7c6538fe446f6b5672b280e8531ee; no voters:
I20260810 03:40:59.078047 1220 raft_consensus.cc:2781] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:40:59.078126 1220 raft_consensus.cc:482] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:40:59.078157 1220 raft_consensus.cc:3037] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:59.079221 1220 raft_consensus.cc:504] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8d908e49d141dd99db70146a0a848b" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43795 } } peers { permanent_uuid: "eaf7c6538fe446f6b5672b280e8531ee" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 46609 } } peers { permanent_uuid: "b080dde8b76544179658d7fb61cde866" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 44667 } }
I20260810 03:40:59.079555 1220 leader_election.cc:288] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [CANDIDATE]: Term 1 election: Requested vote from peers 9a8d908e49d141dd99db70146a0a848b (127.21.175.65:43795), b080dde8b76544179658d7fb61cde866 (127.21.175.67:44667)
I20260810 03:40:59.080024 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc21c2d3b5c4a75b386bcf07e4e7424" candidate_uuid: "eaf7c6538fe446f6b5672b280e8531ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a8d908e49d141dd99db70146a0a848b"
I20260810 03:40:59.080193 844 raft_consensus.cc:3037] T ccc21c2d3b5c4a75b386bcf07e4e7424 P 9a8d908e49d141dd99db70146a0a848b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:59.081431 844 raft_consensus.cc:2454] T ccc21c2d3b5c4a75b386bcf07e4e7424 P 9a8d908e49d141dd99db70146a0a848b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf7c6538fe446f6b5672b280e8531ee in term 1.
I20260810 03:40:59.081858 919 leader_election.cc:302] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a8d908e49d141dd99db70146a0a848b, eaf7c6538fe446f6b5672b280e8531ee; no voters:
I20260810 03:40:59.081832 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc21c2d3b5c4a75b386bcf07e4e7424" candidate_uuid: "eaf7c6538fe446f6b5672b280e8531ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b080dde8b76544179658d7fb61cde866"
I20260810 03:40:59.081995 1143 raft_consensus.cc:3037] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:40:59.082068 1220 raft_consensus.cc:2781] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:40:59.082301 1220 raft_consensus.cc:686] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [term 1 LEADER]: Becoming Leader. State: Replica: eaf7c6538fe446f6b5672b280e8531ee, State: Running, Role: LEADER
I20260810 03:40:59.082757 1220 consensus_queue.cc:237] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8d908e49d141dd99db70146a0a848b" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43795 } } peers { permanent_uuid: "eaf7c6538fe446f6b5672b280e8531ee" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 46609 } } peers { permanent_uuid: "b080dde8b76544179658d7fb61cde866" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 44667 } }
I20260810 03:40:59.083498 1143 raft_consensus.cc:2454] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf7c6538fe446f6b5672b280e8531ee in term 1.
I20260810 03:40:59.086735 686 catalog_manager.cc:5495] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf7c6538fe446f6b5672b280e8531ee (127.21.175.66). New cstate: current_term: 1 leader_uuid: "eaf7c6538fe446f6b5672b280e8531ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8d908e49d141dd99db70146a0a848b" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf7c6538fe446f6b5672b280e8531ee" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 46609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b080dde8b76544179658d7fb61cde866" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 44667 } health_report { overall_health: UNKNOWN } } }
W20260810 03:40:59.099999 1196 tablet.cc:2369] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:40:59.106622 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:59.102519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.21.175.66:46609)
0810 03:40:59.102677 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 03:40:59.102680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:40:59.102693 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:40:59.102780 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:40:59.102783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:40:59.102875 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:40:59.102976 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:40:59.103290 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:59.103855 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:59.104847 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:40:59.105093 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:40:59.105181 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:40:59.105668 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:40:59.106132 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:40:59.106248 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 03:40:59.106372 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 03:40:59.106437 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:40:59.113404 1143 raft_consensus.cc:1261] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866 [term 1 FOLLOWER]: Refusing update from remote peer eaf7c6538fe446f6b5672b280e8531ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:59.113420 844 raft_consensus.cc:1261] T ccc21c2d3b5c4a75b386bcf07e4e7424 P 9a8d908e49d141dd99db70146a0a848b [term 1 FOLLOWER]: Refusing update from remote peer eaf7c6538fe446f6b5672b280e8531ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:40:59.114315 1220 consensus_queue.cc:1035] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "b080dde8b76544179658d7fb61cde866" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 44667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:40:59.114542 1224 consensus_queue.cc:1035] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a8d908e49d141dd99db70146a0a848b" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 03:40:59.114959 1048 tablet.cc:2369] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:40:59.125000 1228 mvcc.cc:205] Tried to move back new op lower bound from 7316821029319548928 to 7316821029207040000. Current Snapshot: MvccSnapshot[applied={T|T < 7316821029319548928 or (T in {7316821029319548928})}]
I20260810 03:40:59.133075 686 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:59.133239 686 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:40:59.135967 686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e4cb5a6610c4508ad98922c0808b257: Sending DeleteTablet for 3 replicas of tablet ccc21c2d3b5c4a75b386bcf07e4e7424
I20260810 03:40:59.137032 822 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc21c2d3b5c4a75b386bcf07e4e7424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:59 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:40850
I20260810 03:40:59.137032 1120 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc21c2d3b5c4a75b386bcf07e4e7424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:59 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:35034
I20260810 03:40:59.137087 968 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc21c2d3b5c4a75b386bcf07e4e7424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:40:59 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:59968
I20260810 03:40:59.137758 1238 tablet_replica.cc:330] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee: stopping tablet replica
I20260810 03:40:59.138019 1236 tablet_replica.cc:330] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866: stopping tablet replica
I20260810 03:40:59.138109 1238 raft_consensus.cc:2229] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:40:59.138327 1236 raft_consensus.cc:2229] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:40:59.138666 1236 raft_consensus.cc:2258] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:59.138785 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 753
I20260810 03:40:59.143172 1238 raft_consensus.cc:2258] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:40:59.145004 1238 ts_tablet_manager.cc:1905] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:59.145005 1236 ts_tablet_manager.cc:1905] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:40:59.147886 1236 ts_tablet_manager.cc:1918] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:40:59.147886 1238 ts_tablet_manager.cc:1918] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:40:59.148010 1238 log.cc:1196] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/ccc21c2d3b5c4a75b386bcf07e4e7424
I20260810 03:40:59.148005 1236 log.cc:1196] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/ccc21c2d3b5c4a75b386bcf07e4e7424
I20260810 03:40:59.148350 1238 ts_tablet_manager.cc:1939] T ccc21c2d3b5c4a75b386bcf07e4e7424 P eaf7c6538fe446f6b5672b280e8531ee: Deleting consensus metadata
I20260810 03:40:59.148350 1236 ts_tablet_manager.cc:1939] T ccc21c2d3b5c4a75b386bcf07e4e7424 P b080dde8b76544179658d7fb61cde866: Deleting consensus metadata
I20260810 03:40:59.149830 667 catalog_manager.cc:4841] TS b080dde8b76544179658d7fb61cde866 (127.21.175.67:44667): tablet ccc21c2d3b5c4a75b386bcf07e4e7424 (table test-table [id=7333432c1cf54c4fb476e55755b7d0dc]) successfully deleted
I20260810 03:40:59.149830 670 catalog_manager.cc:4841] TS eaf7c6538fe446f6b5672b280e8531ee (127.21.175.66:46609): tablet ccc21c2d3b5c4a75b386bcf07e4e7424 (table test-table [id=7333432c1cf54c4fb476e55755b7d0dc]) successfully deleted
W20260810 03:40:59.150206 670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.175.65:43795 (error 108)
I20260810 03:40:59.150480 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 901
I20260810 03:40:59.154891 1210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:40:59.154580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:43795 (local address 127.0.0.1:40852)
0810 03:40:59.154726 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 03:40:59.154804 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:43795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 03:40:59.155068 670 catalog_manager.cc:4584] TS 9a8d908e49d141dd99db70146a0a848b (127.21.175.65:43795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ccc21c2d3b5c4a75b386bcf07e4e7424: Network error: Client connection negotiation failed: client connection to 127.21.175.65:43795: connect: Connection refused (error 111)
I20260810 03:40:59.159965 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 1052
I20260810 03:40:59.168256 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 651
2026-08-10T03:40:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2658 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:40:59 dist-test-slave-ss4n krb5kdc[1242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:40:59 dist-test-slave-ss4n krb5kdc[1242](info): set up 2 sockets
Aug 10 03:40:59 dist-test-slave-ss4n krb5kdc[1242](info): commencing operation
krb5kdc: starting...
W20260810 03:41:00.330294 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:41:00 dist-test-slave-ss4n krb5kdc[1242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:41:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:41:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.175.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.175.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.175.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/rangeradmin_127.21.175.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.175.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/rangeradmin_127.21.175.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.175.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.175.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.175.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/rangerlookup_127.21.175.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.175.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/rangerlookup_127.21.175.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.175.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.175.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.175.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/HTTP_127.21.175.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.175.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/HTTP_127.21.175.84@KRBTEST.COM.keytab.
I20260810 03:41:00.533640 22205 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMmCxq6/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/postgres -l logfile start
2026-08-10 03:41:03.367 UTC [1273] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 03:41:03.368 UTC [1273] LOG: listening on IPv4 address "127.21.175.84", port 53227
2026-08-10 03:41:03.372 UTC [1273] LOG: listening on Unix socket "/tmp/.s.PGSQL.53227"
2026-08-10 03:41:03.412 UTC [1277] LOG: database system was shut down at 2026-08-10 03:41:01 UTC
2026-08-10 03:41:03.417 UTC [1273] LOG: database system is ready to accept connections
I20260810 03:41:04.430979 22205 mini_postgres.cc:96] Postgres bound to 53227
2026-08-10 03:41:04.439 UTC [1287] FATAL: database "slave" does not exist
127.21.175.84:53227 - accepting connections
I20260810 03:41:04.440066 22205 mini_ranger.cc:161] Starting Ranger...
I20260810 03:41:04.456390 22205 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 03:41:05.045070 22205 mini_ranger.cc:87] Created ranger Postgres database
I20260810 03:41:05.045198 22205 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ranger-admin
2026-08-10 03:41:05,373 [I] DB FLAVOR :POSTGRES
2026-08-10 03:41:05,373 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 03:41:05,374 [I] Checking connection..
2026-08-10 03:41:05,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 03:41:05,632 [I] Checking connection passed.
2026-08-10 03:41:05,633 [I] --------- Verifying version history table ---------
2026-08-10 03:41:05,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:41:05,891 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 03:41:05,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:41:06,168 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 03:41:06,168 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 03:41:06,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 03:41:06.424 UTC [1358] WARNING: there is no transaction in progress
2026-08-10 03:41:06,432 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 03:41:06,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:41:06,726 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 03:41:06,727 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 03:41:06,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 03:41:07,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 03:41:07,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ss4n.c.gcp-upstream.internal','N') ;"
2026-08-10 03:41:07,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 03:41:07,776 [I] Table x_portal_user does not exist in database ranger
2026-08-10 03:41:07,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 03:41:08,069 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 03:41:08,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 03:41:08,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 03:41:08,606 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 03:41:08,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 03:41:09.720 UTC [1471] WARNING: there is no transaction in progress
2026-08-10 03:41:09.740 UTC [1471] WARNING: there is no transaction in progress
2026-08-10 03:41:09.760 UTC [1471] WARNING: there is no transaction in progress
2026-08-10 03:41:09.779 UTC [1471] WARNING: there is no transaction in progress
2026-08-10 03:41:09.802 UTC [1471] WARNING: there is no transaction in progress
2026-08-10 03:41:09.826 UTC [1471] WARNING: there is no transaction in progress
2026-08-10 03:41:09.889 UTC [1471] WARNING: there is no transaction in progress
2026-08-10 03:41:09.898 UTC [1471] WARNING: there is no transaction in progress
2026-08-10 03:41:09.914 UTC [1471] WARNING: there is no transaction in progress
2026-08-10 03:41:09.927 UTC [1471] WARNING: there is no transaction in progress
2026-08-10 03:41:09.938 UTC [1471] WARNING: there is no transaction in progress
2026-08-10 03:41:09.954 UTC [1471] WARNING: there is no transaction in progress
2026-08-10 03:41:09.965 UTC [1471] WARNING: there is no transaction in progress
2026-08-10 03:41:09.972 UTC [1471] WARNING: there is no transaction in progress
2026-08-10 03:41:09.980 UTC [1471] WARNING: there is no transaction in progress
2026-08-10 03:41:09.988 UTC [1471] WARNING: there is no transaction in progress
2026-08-10 03:41:10,764 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 03:41:10,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 03:41:11,036 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 03:41:11,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 03:41:11,310 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 03:41:11,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 03:41:11,581 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 03:41:11,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 03:41:11,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 03:41:12,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ss4n.c.gcp-upstream.internal';"
2026-08-10 03:41:12,402 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 03:41:12,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 03:41:12,669 [I] DB_PATCHES have already been applied
I20260810 03:41:12.675531 22205 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMmCxq6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 03:41:12.675617 22205 mini_ranger.cc:193] Using host: 127.21.175.84
I20260810 03:41:12.680686 22205 mini_ranger.cc:227] Ranger admin URL: http://127.21.175.84:37087
Aug 10, 2026 3:41:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:41:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:41:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 3:41:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 3:41:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 3:41:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 3:41:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 03:41:14.010663 22205 mini_ranger.cc:160] Time spent starting Ranger: real 9.571s user 0.000s sys 0.014s
Aug 10, 2026 3:41:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 3:41:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 10, 2026 3:41:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 3:41:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 3:41:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 3:41:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 3:41:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 3:41:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 3:41:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 3:41:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 3:41:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 3:41:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 3:41:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 3:41:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 3:41:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 3:41:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 3:41:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 3:41:35 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 03:41:36.150168 22205 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.175.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.175.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.175.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/rangerkms_127.21.175.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.175.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/rangerkms_127.21.175.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.175.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.175.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.175.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/HTTP_127.21.175.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.175.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/HTTP_127.21.175.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 03:41:36 dist-test-slave-ss4n krb5kdc[1242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333296, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 03:41:36.259466 22205 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 03:41:36.270687 22205 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 03:41:37.367712 22205 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 03:41:37.367833 22205 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ranger-kms
I20260810 03:41:37.367965 22205 mini_ranger_kms.cc:227] Using postgres at 127.21.175.84:53227
2026-08-10 03:41:37,900 [I] DB FLAVOR :POSTGRES
2026-08-10 03:41:37,900 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 03:41:37,900 [I] Checking connection
2026-08-10 03:41:37,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 03:41:38,177 [I] connection success
2026-08-10 03:41:38,178 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 03:41:38,178 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 03:41:38,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 03:41:38,450 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 03:41:38,451 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 03:41:38,451 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 03:41:38,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:53227/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 03:41:38,725 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 03:41:39.249908 22205 mini_ranger_kms.cc:310] Created kms service
I20260810 03:41:39.468477 22205 mini_ranger_kms.cc:326] Created kudu user
I20260810 03:41:39.526125 22205 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 03:41:39 dist-test-slave-ss4n krb5kdc[1242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333296, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
I20260810 03:41:39.950692 22205 mini_ranger_kms.cc:384] Added ranger policy
I20260810 03:41:39.951041 22205 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 03:41:39.951064 22205 mini_ranger_kms.cc:240] Using host: 127.21.175.84
I20260810 03:41:39.956341 22205 mini_ranger_kms.cc:276] Ranger KMS PID: 1670
I20260810 03:41:39.956604 22205 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.175.84:45829
Aug 10, 2026 3:41:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:41:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:41:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 3:41:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 3:41:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 3:41:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260810 03:41:42.467943 22205 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.208s user 0.021s sys 0.278s
I20260810 03:41:42.468127 22205 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 03:41:42.468272 22205 mini_ranger_kms.cc:401] 127.21.175.84:45829/kms/v1/keys
Aug 10 03:41:42 dist-test-slave-ss4n krb5kdc[1242](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786333302, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.175.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 03:41:43 dist-test-slave-ss4n krb5kdc[1242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333302, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.175.84@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
Aug 10 03:41:43 dist-test-slave-ss4n krb5kdc[1242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333302, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.175.84@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
Aug 10 03:41:45 dist-test-slave-ss4n krb5kdc[1242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:41:45.941830 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:44219
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:44219
--ranger_config_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.175.84:45829/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:41:46.061338 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:41:46.061782 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:41:46.061874 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:41:46.066112 1723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:41:46.066238 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:41:46.066278 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:41:46.066315 1723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:41:46.066351 1723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:41:46.072324 1723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.175.84:45829/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:44219
--ranger_config_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:44219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:41:46.073597 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:41:46.074716 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:41:46.084534 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:41:46.084573 1723 server_base.cc:1034] running on GCE node
W20260810 03:41:46.084610 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:41:46.084864 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:41:46.085541 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:41:46.086890 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:41:46.088093 1723 hybrid_clock.cc:648] HybridClock initialized: now 1786333306088080 us; error 30 us; skew 500 ppm
Aug 10 03:41:46 dist-test-slave-ss4n krb5kdc[1242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333306, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:41:46.091156 1723 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:41:46.092061 1723 webserver.cc:460] Webserver started at http://127.21.175.126:44695/ using document root <none> and password file <none>
I20260810 03:41:46.092602 1723 fs_manager.cc:359] Metadata directory not provided
I20260810 03:41:46.092664 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:41:46.092839 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:41:46 dist-test-slave-ss4n krb5kdc[1242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333306, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
I20260810 03:41:46.155229 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "375788d140f248fb9a00b8463a4ca518"
format_stamp: "Formatted at 2026-08-10 03:41:46 on dist-test-slave-ss4n"
server_key: "6d7283f747b306fb974aecad0983c2a5"
server_key_iv: "53c2bf7dc0a1be617a2a6c2ccfd7846f"
server_key_version: "kuduclusterkey@0"
I20260810 03:41:46.156071 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "375788d140f248fb9a00b8463a4ca518"
format_stamp: "Formatted at 2026-08-10 03:41:46 on dist-test-slave-ss4n"
server_key: "6d7283f747b306fb974aecad0983c2a5"
server_key_iv: "53c2bf7dc0a1be617a2a6c2ccfd7846f"
server_key_version: "kuduclusterkey@0"
I20260810 03:41:46.162127 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260810 03:41:46.178603 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:41:46.180297 1723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:41:46.180429 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "375788d140f248fb9a00b8463a4ca518"
format_stamp: "Formatted at 2026-08-10 03:41:46 on dist-test-slave-ss4n"
server_key: "6d7283f747b306fb974aecad0983c2a5"
server_key_iv: "53c2bf7dc0a1be617a2a6c2ccfd7846f"
server_key_version: "kuduclusterkey@0"
I20260810 03:41:46.180619 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:41:46.197970 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:41:46.201656 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:41:46.201896 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:41:46.210114 1723 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:44219
I20260810 03:41:46.210141 1795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:44219 every 8 connection(s)
I20260810 03:41:46.211158 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:41:46.211807 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 1723
I20260810 03:41:46.211957 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:41:46.212265 22205 external_mini_cluster.cc:1428] Setting key 4758a9dd6d992cd1bd60c68723a9e88f
I20260810 03:41:46.214437 1796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:41:46.221469 1796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518: Bootstrap starting.
Aug 10 03:41:46 dist-test-slave-ss4n krb5kdc[1242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:41:46.224467 1796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518: Neither blocks nor log segments found. Creating new log.
I20260810 03:41:46.225903 1796 log.cc:824] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518: Log is configured to *not* fsync() on all Append() calls
I20260810 03:41:46.228372 1796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518: No bootstrap required, opened a new log
I20260810 03:41:46.228753 1801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:46.213804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37632 (local address 127.21.175.126:44219)
0810 03:41:46.214251 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0810 03:41:46.214267 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:41:46.214303 (+ 36us) server_negotiation.cc:407] Connection header received
0810 03:41:46.214985 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:41:46.215012 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:41:46.215341 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:41:46.215635 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:41:46.216779 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:46.218098 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:46.219353 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:46.219870 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:46.220004 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:46.223244 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:41:46.223251 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:41:46.223268 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:41:46.223303 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:41:46.225442 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:41:46.226008 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:41:46.226017 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:41:46.226024 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:41:46.226152 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:41:46.226491 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:41:46.226494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:41:46.226495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:41:46.226965 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:41:46.227151 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 03:41:46.227469 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 03:41:46.227760 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":146,"threads_started":1}
I20260810 03:41:46.231990 1796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375788d140f248fb9a00b8463a4ca518" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44219 } }
I20260810 03:41:46.232219 1796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:41:46.232256 1796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375788d140f248fb9a00b8463a4ca518, State: Initialized, Role: FOLLOWER
I20260810 03:41:46.232668 1796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375788d140f248fb9a00b8463a4ca518" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44219 } }
I20260810 03:41:46.232792 1796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:41:46.232832 1796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:41:46.232895 1796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:41:46.233876 1796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375788d140f248fb9a00b8463a4ca518" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44219 } }
I20260810 03:41:46.234185 1796 leader_election.cc:302] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 375788d140f248fb9a00b8463a4ca518; no voters:
I20260810 03:41:46.234450 1796 leader_election.cc:288] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:41:46.234639 1803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:41:46.235113 1803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [term 1 LEADER]: Becoming Leader. State: Replica: 375788d140f248fb9a00b8463a4ca518, State: Running, Role: LEADER
I20260810 03:41:46.235515 1803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375788d140f248fb9a00b8463a4ca518" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44219 } }
I20260810 03:41:46.235826 1796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:41:46.237365 1805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 375788d140f248fb9a00b8463a4ca518. Latest consensus state: current_term: 1 leader_uuid: "375788d140f248fb9a00b8463a4ca518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375788d140f248fb9a00b8463a4ca518" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44219 } } }
I20260810 03:41:46.237561 1805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [sys.catalog]: This master's current role is: LEADER
I20260810 03:41:46.237941 1804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "375788d140f248fb9a00b8463a4ca518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375788d140f248fb9a00b8463a4ca518" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44219 } } }
I20260810 03:41:46.238054 1804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518 [sys.catalog]: This master's current role is: LEADER
I20260810 03:41:46.238224 1796 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 03:41:46.238938 1816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:41:46.245617 1816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:41:46.253670 1816 catalog_manager.cc:1347] Generated new cluster ID: 30925947d68946209fc7baa48ec2ed46
I20260810 03:41:46.253753 1816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:41:46.262728 1816 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:41:46.264071 1816 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:41:46.281265 1816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518: Generated new TSK 0
I20260810 03:41:46.282200 1816 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:41:47.058169 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.175.84:45829/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:44219
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:41:47.192543 1845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:41:47.192866 1845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:41:47.193027 1845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:41:47.197867 1845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:41:47.198047 1845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:41:47.198199 1845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:41:47.205905 1845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.175.84:45829/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:44219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:41:47.207283 1845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:41:47.208557 1845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:41:47.220897 1855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:41:47.220922 1852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:41:47.221406 1845 server_base.cc:1034] running on GCE node
W20260810 03:41:47.222252 1853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:41:47.222844 1845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:41:47.223737 1845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:41:47.227035 1845 hybrid_clock.cc:648] HybridClock initialized: now 1786333307226986 us; error 60 us; skew 500 ppm
Aug 10 03:41:47 dist-test-slave-ss4n krb5kdc[1242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333307, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:41:47.231132 1845 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:41:47.232458 1845 webserver.cc:460] Webserver started at http://127.21.175.65:43267/ using document root <none> and password file <none>
I20260810 03:41:47.233247 1845 fs_manager.cc:359] Metadata directory not provided
I20260810 03:41:47.233325 1845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:41:47.233656 1845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:41:47 dist-test-slave-ss4n krb5kdc[1242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333307, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
I20260810 03:41:47.254416 1845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "8ff4cd659f1d4cf59e16674efd5a53cc"
format_stamp: "Formatted at 2026-08-10 03:41:47 on dist-test-slave-ss4n"
server_key: "fc376f5a2b9420328d763c8a89d294ac"
server_key_iv: "20a0bc42f2051e1fb932f1ab429dabb5"
server_key_version: "kuduclusterkey@0"
I20260810 03:41:47.255179 1845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "8ff4cd659f1d4cf59e16674efd5a53cc"
format_stamp: "Formatted at 2026-08-10 03:41:47 on dist-test-slave-ss4n"
server_key: "fc376f5a2b9420328d763c8a89d294ac"
server_key_iv: "20a0bc42f2051e1fb932f1ab429dabb5"
server_key_version: "kuduclusterkey@0"
I20260810 03:41:47.260115 1845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:41:47.294103 1863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:41:47.296128 1845 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.000s
I20260810 03:41:47.296284 1845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "8ff4cd659f1d4cf59e16674efd5a53cc"
format_stamp: "Formatted at 2026-08-10 03:41:47 on dist-test-slave-ss4n"
server_key: "fc376f5a2b9420328d763c8a89d294ac"
server_key_iv: "20a0bc42f2051e1fb932f1ab429dabb5"
server_key_version: "kuduclusterkey@0"
I20260810 03:41:47.296442 1845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:41:47.327270 1845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:41:47.331568 1845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:41:47.331835 1845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:41:47.332636 1845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:41:47.333909 1845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:41:47.334025 1845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:41:47.334146 1845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:41:47.334208 1845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:41:47.346709 1845 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:41645
I20260810 03:41:47.348124 1845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:41:47.348780 1991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:41645 every 8 connection(s)
I20260810 03:41:47.358284 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 1845
I20260810 03:41:47.358419 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:41:47.358753 22205 external_mini_cluster.cc:1428] Setting key d61d457001be0a18a75c16a0a3f8be86
Aug 10 03:41:47 dist-test-slave-ss4n krb5kdc[1242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333307, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:41:47.367954 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:47.350771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:48169 (local address 127.21.175.126:44219)
0810 03:41:47.351062 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0810 03:41:47.351067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:41:47.352097 (+ 1030us) server_negotiation.cc:407] Connection header received
0810 03:41:47.353585 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:41:47.353590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:41:47.353655 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:41:47.353762 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:41:47.355436 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:47.356327 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:47.358351 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:47.358748 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:47.358848 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:47.362689 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:41:47.362694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:41:47.362697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:41:47.362729 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:41:47.365030 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:41:47.365839 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:41:47.365844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:41:47.365846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:41:47.365953 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:41:47.366417 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:41:47.366421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:41:47.366423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:41:47.366688 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:41:47.366781 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 03:41:47.367582 (+ 801us) server_negotiation.cc:299] Negotiation successful
0810 03:41:47.367741 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
I20260810 03:41:47.369895 1995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:47.351268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:44219 (local address 127.21.175.65:48169)
0810 03:41:47.351893 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0810 03:41:47.351959 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 03:41:47.353252 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:41:47.354003 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:41:47.354014 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:41:47.354450 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:41:47.355220 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:41:47.355247 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:41:47.356502 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:41:47.356506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:41:47.358170 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:41:47.358184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:41:47.359121 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:41:47.359126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:41:47.359364 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:47.360180 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:41:47.360205 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:41:47.362434 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:41:47.365176 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:41:47.365184 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:41:47.365205 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:41:47.365690 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:41:47.366082 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:41:47.366085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:41:47.366087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:41:47.366289 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:41:47.366860 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:41:47.366867 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:41:47.367238 (+ 371us) client_negotiation.cc:769] Sending connection context
0810 03:41:47.368171 (+ 933us) client_negotiation.cc:240] Negotiation successful
0810 03:41:47.368605 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":150,"threads_started":1}
I20260810 03:41:47.371731 1992 heartbeater.cc:347] Connected to a master server at 127.21.175.126:44219
I20260810 03:41:47.372069 1992 heartbeater.cc:464] Registering TS with master...
I20260810 03:41:47.372844 1992 heartbeater.cc:511] Master 127.21.175.126:44219 requested a full tablet report, sending...
I20260810 03:41:47.375000 1755 ts_manager.cc:194] Registered new tserver with Master: 8ff4cd659f1d4cf59e16674efd5a53cc (127.21.175.65:41645)
I20260810 03:41:47.376734 1755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:48169
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:41:47.405732 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.175.84:45829/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:44219
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:41:47 dist-test-slave-ss4n krb5kdc[1242](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786333307, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 03:41:47.590533 1998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:41:47.590926 1998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:41:47.591007 1998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:41:47.597232 1998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:41:47.597328 1998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:41:47.597424 1998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:41:47.606591 1998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.175.84:45829/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:44219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:41:47.608253 1998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:41:47.609812 1998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:41:47.625092 2006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:41:47.625087 2011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:41:47.625962 1998 server_base.cc:1034] running on GCE node
W20260810 03:41:47.626199 2008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:41:47.626813 1998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:41:47.627821 1998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:41:47.629052 1998 hybrid_clock.cc:648] HybridClock initialized: now 1786333307629017 us; error 51 us; skew 500 ppm
Aug 10 03:41:47 dist-test-slave-ss4n krb5kdc[1242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333307, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:41:47.633381 1998 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:41:47.634718 1998 webserver.cc:460] Webserver started at http://127.21.175.66:33537/ using document root <none> and password file <none>
I20260810 03:41:47.635679 1998 fs_manager.cc:359] Metadata directory not provided
I20260810 03:41:47.635830 1998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:41:47.636154 1998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:41:47 dist-test-slave-ss4n krb5kdc[1242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333307, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
I20260810 03:41:47.656018 1998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "91d65c0b8752491985a22e544fc97c7d"
format_stamp: "Formatted at 2026-08-10 03:41:47 on dist-test-slave-ss4n"
server_key: "1a365b676b7aed166e75c0602f06ffa8"
server_key_iv: "ac8c7851789091002dfb89c346c14d56"
server_key_version: "kuduclusterkey@0"
I20260810 03:41:47.656908 1998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "91d65c0b8752491985a22e544fc97c7d"
format_stamp: "Formatted at 2026-08-10 03:41:47 on dist-test-slave-ss4n"
server_key: "1a365b676b7aed166e75c0602f06ffa8"
server_key_iv: "ac8c7851789091002dfb89c346c14d56"
server_key_version: "kuduclusterkey@0"
I20260810 03:41:47.663091 1998 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260810 03:41:47.700009 2019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:41:47.702128 1998 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.000s
I20260810 03:41:47.702327 1998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "91d65c0b8752491985a22e544fc97c7d"
format_stamp: "Formatted at 2026-08-10 03:41:47 on dist-test-slave-ss4n"
server_key: "1a365b676b7aed166e75c0602f06ffa8"
server_key_iv: "ac8c7851789091002dfb89c346c14d56"
server_key_version: "kuduclusterkey@0"
I20260810 03:41:47.702487 1998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:41:47.752125 1998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:41:47.756614 1998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:41:47.756891 1998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:41:47.757726 1998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:41:47.758868 1998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:41:47.758987 1998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:41:47.759083 1998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:41:47.759145 1998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:41:47.772058 1998 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:45987
I20260810 03:41:47.773463 1998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:41:47.775900 2144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:45987 every 8 connection(s)
I20260810 03:41:47.778131 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 1998
I20260810 03:41:47.778280 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:41:47.778646 22205 external_mini_cluster.cc:1428] Setting key 301c714d4150c73c445fea4a052cd582
Aug 10 03:41:47 dist-test-slave-ss4n krb5kdc[1242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333307, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:41:47.796499 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:47.776335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:34189 (local address 127.21.175.126:44219)
0810 03:41:47.776476 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 03:41:47.776481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:41:47.777559 (+ 1078us) server_negotiation.cc:407] Connection header received
0810 03:41:47.779008 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:41:47.779012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:41:47.779072 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:41:47.779130 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:41:47.784815 (+ 5685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:47.785771 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:47.787603 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:47.787976 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:47.788062 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:47.791757 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:41:47.791771 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:41:47.791774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:41:47.791806 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:41:47.793858 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:41:47.794556 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:41:47.794561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:41:47.794562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:41:47.794650 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:41:47.795096 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:41:47.795100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:41:47.795102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:41:47.795333 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:41:47.795409 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 03:41:47.796176 (+ 767us) server_negotiation.cc:299] Negotiation successful
0810 03:41:47.796320 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:41:47.798568 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:47.776825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:44219 (local address 127.21.175.66:34189)
0810 03:41:47.777340 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 03:41:47.777395 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 03:41:47.778659 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:41:47.783294 (+ 4635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:41:47.783305 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:41:47.783704 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:41:47.784626 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:41:47.784650 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:41:47.786014 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:41:47.786017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:41:47.787449 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:41:47.787461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:41:47.788290 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:41:47.788294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:41:47.788505 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:47.789550 (+ 1045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:41:47.789579 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:41:47.791503 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:41:47.794032 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:41:47.794038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:41:47.794058 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:41:47.794424 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:41:47.794759 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:41:47.794762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:41:47.794764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:41:47.794975 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:41:47.795496 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:41:47.795503 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:41:47.795859 (+ 356us) client_negotiation.cc:769] Sending connection context
0810 03:41:47.797023 (+ 1164us) client_negotiation.cc:240] Negotiation successful
0810 03:41:47.797400 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":157,"threads_started":1}
I20260810 03:41:47.800220 2145 heartbeater.cc:347] Connected to a master server at 127.21.175.126:44219
I20260810 03:41:47.800487 2145 heartbeater.cc:464] Registering TS with master...
I20260810 03:41:47.801136 2145 heartbeater.cc:511] Master 127.21.175.126:44219 requested a full tablet report, sending...
I20260810 03:41:47.802671 1755 ts_manager.cc:194] Registered new tserver with Master: 91d65c0b8752491985a22e544fc97c7d (127.21.175.66:45987)
I20260810 03:41:47.803581 1755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:34189
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:41:47.843554 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.175.84:45829/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:44219
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:41:47 dist-test-slave-ss4n krb5kdc[1242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333307, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
W20260810 03:41:48.044471 2151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:41:48.045321 2151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:41:48.045429 2151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:41:48.051671 2151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:41:48.051860 2151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:41:48.052008 2151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:41:48.060372 2151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.175.84:45829/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:44219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:41:48.062223 2151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:41:48.063691 2151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:41:48.080294 2159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:41:48.080747 2157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:41:48.080287 2156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:41:48.080880 2151 server_base.cc:1034] running on GCE node
I20260810 03:41:48.081439 2151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:41:48.082309 2151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:41:48.083499 2151 hybrid_clock.cc:648] HybridClock initialized: now 1786333308083429 us; error 87 us; skew 500 ppm
Aug 10 03:41:48 dist-test-slave-ss4n krb5kdc[1242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333308, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:41:48.093029 2151 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:41:48.094348 2151 webserver.cc:460] Webserver started at http://127.21.175.67:37887/ using document root <none> and password file <none>
I20260810 03:41:48.095160 2151 fs_manager.cc:359] Metadata directory not provided
I20260810 03:41:48.095309 2151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:41:48.095614 2151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:41:48 dist-test-slave-ss4n krb5kdc[1242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333308, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
I20260810 03:41:48.116848 2151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "a24d26e24e504db5b96f1164e329d1c6"
format_stamp: "Formatted at 2026-08-10 03:41:48 on dist-test-slave-ss4n"
server_key: "42f3402581ee80804636d85a41f3922e"
server_key_iv: "4eae7be01c1b35c25123bf9b1749bb4e"
server_key_version: "kuduclusterkey@0"
I20260810 03:41:48.117758 2151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "a24d26e24e504db5b96f1164e329d1c6"
format_stamp: "Formatted at 2026-08-10 03:41:48 on dist-test-slave-ss4n"
server_key: "42f3402581ee80804636d85a41f3922e"
server_key_iv: "4eae7be01c1b35c25123bf9b1749bb4e"
server_key_version: "kuduclusterkey@0"
I20260810 03:41:48.130755 2151 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.000s
I20260810 03:41:48.153832 2167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:41:48.155769 2151 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260810 03:41:48.155929 2151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "a24d26e24e504db5b96f1164e329d1c6"
format_stamp: "Formatted at 2026-08-10 03:41:48 on dist-test-slave-ss4n"
server_key: "42f3402581ee80804636d85a41f3922e"
server_key_iv: "4eae7be01c1b35c25123bf9b1749bb4e"
server_key_version: "kuduclusterkey@0"
I20260810 03:41:48.156098 2151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:41:48.184197 2151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:41:48.188445 2151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:41:48.188745 2151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:41:48.189651 2151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:41:48.190912 2151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:41:48.191048 2151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:41:48.191195 2151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:41:48.191288 2151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:41:48.207566 2151 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:46055
I20260810 03:41:48.209120 2151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:41:48.217119 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 2151
I20260810 03:41:48.217275 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:41:48.217794 22205 external_mini_cluster.cc:1428] Setting key 68d96a0fabc4aaaa6c1cf2706bd9b804
I20260810 03:41:48.223923 2292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:46055 every 8 connection(s)
Aug 10 03:41:48 dist-test-slave-ss4n krb5kdc[1242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333308, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:41:48.246589 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:48.223226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:41307 (local address 127.21.175.126:44219)
0810 03:41:48.223372 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 03:41:48.223377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:41:48.230231 (+ 6854us) server_negotiation.cc:407] Connection header received
0810 03:41:48.231782 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:41:48.231787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:41:48.231847 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:41:48.231902 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:41:48.234099 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.235043 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.237231 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.237675 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.237778 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:48.241463 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:41:48.241468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:41:48.241471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:41:48.241547 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:41:48.243644 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:41:48.244431 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:41:48.244436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:41:48.244438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:41:48.244536 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:41:48.245051 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:41:48.245055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:41:48.245057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:41:48.245313 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:41:48.245395 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 03:41:48.246259 (+ 864us) server_negotiation.cc:299] Negotiation successful
0810 03:41:48.246409 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:41:48.248764 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:48.221741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:44219 (local address 127.21.175.67:41307)
0810 03:41:48.229942 (+ 8201us) negotiation.cc:107] Waiting for socket to connect
0810 03:41:48.230041 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0810 03:41:48.231468 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:41:48.232524 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:41:48.232536 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:41:48.233020 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:41:48.233878 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:41:48.233901 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.235228 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.235232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:41:48.237065 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:41:48.237078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.238029 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.238034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:41:48.238274 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:48.239177 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:41:48.239204 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:41:48.241232 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:41:48.243844 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:41:48.243852 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:41:48.243872 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:41:48.244294 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:41:48.244667 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:41:48.244670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:41:48.244672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:41:48.244901 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:41:48.245534 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:41:48.245541 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:41:48.245920 (+ 379us) client_negotiation.cc:769] Sending connection context
0810 03:41:48.247121 (+ 1201us) client_negotiation.cc:240] Negotiation successful
0810 03:41:48.247532 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7880,"thread_start_us":253,"threads_started":1}
I20260810 03:41:48.250638 2293 heartbeater.cc:347] Connected to a master server at 127.21.175.126:44219
I20260810 03:41:48.250926 2293 heartbeater.cc:464] Registering TS with master...
I20260810 03:41:48.251624 2293 heartbeater.cc:511] Master 127.21.175.126:44219 requested a full tablet report, sending...
I20260810 03:41:48.252964 1755 ts_manager.cc:194] Registered new tserver with Master: a24d26e24e504db5b96f1164e329d1c6 (127.21.175.67:46055)
I20260810 03:41:48.253906 1755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:41307
I20260810 03:41:48.255188 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:41:48.267985 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:48.257473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37638 (local address 127.21.175.126:44219)
0810 03:41:48.257610 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 03:41:48.257615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:41:48.258192 (+ 577us) server_negotiation.cc:407] Connection header received
0810 03:41:48.258376 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:41:48.258380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:41:48.258427 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:41:48.258484 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:41:48.259530 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.260304 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.261548 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.261867 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.261955 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:48.264154 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:41:48.264158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:41:48.264161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:41:48.264190 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:41:48.266031 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:41:48.266543 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:41:48.266547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:41:48.266549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:41:48.266636 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:41:48.266953 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:41:48.266956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:41:48.266958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:41:48.267174 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:41:48.267280 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 03:41:48.267667 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 03:41:48.267797 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:41:48.272388 1754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:41:48.277117 1754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:41:48.319288 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:48.290139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:45987 (local address 127.0.0.1:48020)
0810 03:41:48.295937 (+ 5798us) negotiation.cc:107] Waiting for socket to connect
0810 03:41:48.296001 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 03:41:48.296144 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:41:48.300257 (+ 4113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:41:48.300264 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:41:48.300288 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:41:48.300446 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:41:48.300455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.316153 (+ 15698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.316157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:41:48.317374 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:41:48.317386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.318829 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.318833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:41:48.318937 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:48.318959 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 03:41:48.319034 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 03:41:48.319116 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5580,"thread_start_us":125,"threads_started":1}
I20260810 03:41:48.319793 2306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:48.293963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:41645 (local address 127.0.0.1:39034)
0810 03:41:48.298402 (+ 4439us) negotiation.cc:107] Waiting for socket to connect
0810 03:41:48.298416 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:41:48.298514 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:41:48.300581 (+ 2067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:41:48.300583 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:41:48.300602 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:41:48.300686 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:41:48.300692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.310388 (+ 9696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.310392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:41:48.311626 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:41:48.311638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.313131 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.313135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:41:48.313264 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:48.313324 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 03:41:48.313456 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 03:41:48.319663 (+ 6207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4376,"thread_start_us":80,"threads_started":1}
I20260810 03:41:48.319975 2310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:48.292727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:46055 (local address 127.0.0.1:54876)
0810 03:41:48.296630 (+ 3903us) negotiation.cc:107] Waiting for socket to connect
0810 03:41:48.296644 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:41:48.296740 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:41:48.301887 (+ 5147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:41:48.301891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:41:48.301908 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:41:48.302013 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:41:48.302020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.306635 (+ 4615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.306639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:41:48.308194 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:41:48.308206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.314829 (+ 6623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.314833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:41:48.314947 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:48.314971 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:41:48.315047 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 03:41:48.319870 (+ 4823us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3842,"thread_start_us":83,"threads_started":1}
I20260810 03:41:48.321363 2311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:48.294356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.21.175.65:41645)
0810 03:41:48.297004 (+ 2648us) server_negotiation.cc:206] Beginning negotiation
0810 03:41:48.297011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:41:48.298618 (+ 1607us) server_negotiation.cc:407] Connection header received
0810 03:41:48.298704 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:41:48.298709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:41:48.298930 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:41:48.299108 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:41:48.302273 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.309322 (+ 7049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.311804 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.312757 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.312871 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:48.313060 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 03:41:48.321075 (+ 8015us) server_negotiation.cc:299] Negotiation successful
0810 03:41:48.321217 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2525,"thread_start_us":90,"threads_started":1}
I20260810 03:41:48.321964 2309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:48.293139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.21.175.67:46055)
0810 03:41:48.296480 (+ 3341us) server_negotiation.cc:206] Beginning negotiation
0810 03:41:48.296494 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:41:48.297062 (+ 568us) server_negotiation.cc:407] Connection header received
0810 03:41:48.297157 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:41:48.297163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:41:48.297414 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:41:48.301718 (+ 4304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:41:48.302821 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.306485 (+ 3664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.313590 (+ 7105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.314495 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.314611 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:48.314764 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 03:41:48.321662 (+ 6898us) server_negotiation.cc:299] Negotiation successful
0810 03:41:48.321797 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3219,"thread_start_us":148,"threads_started":1}
I20260810 03:41:48.323514 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:48.291181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48020 (local address 127.21.175.66:45987)
0810 03:41:48.299583 (+ 8402us) server_negotiation.cc:206] Beginning negotiation
0810 03:41:48.299600 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:41:48.299623 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:41:48.299699 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:41:48.299704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:41:48.299885 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:41:48.300068 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:41:48.303277 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.309142 (+ 5865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.317582 (+ 8440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.318499 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.318613 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:48.318766 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 03:41:48.323233 (+ 4467us) server_negotiation.cc:299] Negotiation successful
0810 03:41:48.323364 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8291,"thread_start_us":128,"threads_started":1}
I20260810 03:41:48.327307 1919 tablet_service.cc:1467] Processing CreateTablet for tablet 646a97d534284d0686f405316d4da159 (DEFAULT_TABLE table=test-table [id=022a255f115d4cd2b5ff0ade15e9dc45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:41:48.328749 1919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646a97d534284d0686f405316d4da159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:41:48.339781 2219 tablet_service.cc:1467] Processing CreateTablet for tablet 646a97d534284d0686f405316d4da159 (DEFAULT_TABLE table=test-table [id=022a255f115d4cd2b5ff0ade15e9dc45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:41:48.342499 2219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646a97d534284d0686f405316d4da159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:41:48.353981 2052 tablet_service.cc:1467] Processing CreateTablet for tablet 646a97d534284d0686f405316d4da159 (DEFAULT_TABLE table=test-table [id=022a255f115d4cd2b5ff0ade15e9dc45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:41:48.356889 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646a97d534284d0686f405316d4da159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:41:48.368033 2314 tablet_bootstrap.cc:492] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d: Bootstrap starting.
I20260810 03:41:48.372285 2314 tablet_bootstrap.cc:654] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d: Neither blocks nor log segments found. Creating new log.
I20260810 03:41:48.373560 2314 log.cc:824] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d: Log is configured to *not* fsync() on all Append() calls
I20260810 03:41:48.376166 2314 tablet_bootstrap.cc:492] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d: No bootstrap required, opened a new log
I20260810 03:41:48.376387 2314 ts_tablet_manager.cc:1397] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 03:41:48.380155 1992 heartbeater.cc:503] Master 127.21.175.126:44219 was elected leader, sending a full tablet report...
I20260810 03:41:48.392030 2317 tablet_bootstrap.cc:492] T 646a97d534284d0686f405316d4da159 P 8ff4cd659f1d4cf59e16674efd5a53cc: Bootstrap starting.
I20260810 03:41:48.393741 2314 raft_consensus.cc:348] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d65c0b8752491985a22e544fc97c7d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45987 } } peers { permanent_uuid: "a24d26e24e504db5b96f1164e329d1c6" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46055 } } peers { permanent_uuid: "8ff4cd659f1d4cf59e16674efd5a53cc" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41645 } }
I20260810 03:41:48.394093 2314 raft_consensus.cc:374] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:41:48.394156 2314 raft_consensus.cc:729] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d65c0b8752491985a22e544fc97c7d, State: Initialized, Role: FOLLOWER
I20260810 03:41:48.394685 2317 tablet_bootstrap.cc:654] T 646a97d534284d0686f405316d4da159 P 8ff4cd659f1d4cf59e16674efd5a53cc: Neither blocks nor log segments found. Creating new log.
I20260810 03:41:48.394695 2314 consensus_queue.cc:260] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d65c0b8752491985a22e544fc97c7d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45987 } } peers { permanent_uuid: "a24d26e24e504db5b96f1164e329d1c6" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46055 } } peers { permanent_uuid: "8ff4cd659f1d4cf59e16674efd5a53cc" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41645 } }
I20260810 03:41:48.395952 2314 ts_tablet_manager.cc:1428] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260810 03:41:48.396142 2317 log.cc:824] T 646a97d534284d0686f405316d4da159 P 8ff4cd659f1d4cf59e16674efd5a53cc: Log is configured to *not* fsync() on all Append() calls
I20260810 03:41:48.396392 2145 heartbeater.cc:503] Master 127.21.175.126:44219 was elected leader, sending a full tablet report...
I20260810 03:41:48.399075 2316 tablet_bootstrap.cc:492] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6: Bootstrap starting.
I20260810 03:41:48.403054 2316 tablet_bootstrap.cc:654] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6: Neither blocks nor log segments found. Creating new log.
I20260810 03:41:48.403513 2317 tablet_bootstrap.cc:492] T 646a97d534284d0686f405316d4da159 P 8ff4cd659f1d4cf59e16674efd5a53cc: No bootstrap required, opened a new log
I20260810 03:41:48.403735 2317 ts_tablet_manager.cc:1397] T 646a97d534284d0686f405316d4da159 P 8ff4cd659f1d4cf59e16674efd5a53cc: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260810 03:41:48.404240 2316 log.cc:824] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6: Log is configured to *not* fsync() on all Append() calls
I20260810 03:41:48.406556 2316 tablet_bootstrap.cc:492] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6: No bootstrap required, opened a new log
I20260810 03:41:48.406781 2316 ts_tablet_manager.cc:1397] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 03:41:48.407824 2317 raft_consensus.cc:348] T 646a97d534284d0686f405316d4da159 P 8ff4cd659f1d4cf59e16674efd5a53cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d65c0b8752491985a22e544fc97c7d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45987 } } peers { permanent_uuid: "a24d26e24e504db5b96f1164e329d1c6" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46055 } } peers { permanent_uuid: "8ff4cd659f1d4cf59e16674efd5a53cc" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41645 } }
I20260810 03:41:48.408119 2317 raft_consensus.cc:374] T 646a97d534284d0686f405316d4da159 P 8ff4cd659f1d4cf59e16674efd5a53cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:41:48.408185 2317 raft_consensus.cc:729] T 646a97d534284d0686f405316d4da159 P 8ff4cd659f1d4cf59e16674efd5a53cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff4cd659f1d4cf59e16674efd5a53cc, State: Initialized, Role: FOLLOWER
I20260810 03:41:48.411266 2316 raft_consensus.cc:348] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d65c0b8752491985a22e544fc97c7d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45987 } } peers { permanent_uuid: "a24d26e24e504db5b96f1164e329d1c6" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46055 } } peers { permanent_uuid: "8ff4cd659f1d4cf59e16674efd5a53cc" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41645 } }
I20260810 03:41:48.411590 2316 raft_consensus.cc:374] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:41:48.411664 2316 raft_consensus.cc:729] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a24d26e24e504db5b96f1164e329d1c6, State: Initialized, Role: FOLLOWER
I20260810 03:41:48.412261 2316 consensus_queue.cc:260] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d65c0b8752491985a22e544fc97c7d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45987 } } peers { permanent_uuid: "a24d26e24e504db5b96f1164e329d1c6" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46055 } } peers { permanent_uuid: "8ff4cd659f1d4cf59e16674efd5a53cc" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41645 } }
I20260810 03:41:48.413441 2316 ts_tablet_manager.cc:1428] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 03:41:48.413690 2317 consensus_queue.cc:260] T 646a97d534284d0686f405316d4da159 P 8ff4cd659f1d4cf59e16674efd5a53cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d65c0b8752491985a22e544fc97c7d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45987 } } peers { permanent_uuid: "a24d26e24e504db5b96f1164e329d1c6" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46055 } } peers { permanent_uuid: "8ff4cd659f1d4cf59e16674efd5a53cc" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41645 } }
I20260810 03:41:48.413887 2293 heartbeater.cc:503] Master 127.21.175.126:44219 was elected leader, sending a full tablet report...
I20260810 03:41:48.414834 2317 ts_tablet_manager.cc:1428] T 646a97d534284d0686f405316d4da159 P 8ff4cd659f1d4cf59e16674efd5a53cc: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
Aug 10 03:41:48 dist-test-slave-ss4n krb5kdc[1242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333307, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
W20260810 03:41:48.474139 2294 tablet.cc:2369] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:41:48.525267 2146 tablet.cc:2369] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:41:48.586148 2323 raft_consensus.cc:482] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:41:48.586444 2323 raft_consensus.cc:504] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d65c0b8752491985a22e544fc97c7d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45987 } } peers { permanent_uuid: "a24d26e24e504db5b96f1164e329d1c6" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46055 } } peers { permanent_uuid: "8ff4cd659f1d4cf59e16674efd5a53cc" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41645 } }
I20260810 03:41:48.588044 2323 leader_election.cc:288] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d65c0b8752491985a22e544fc97c7d (127.21.175.66:45987), 8ff4cd659f1d4cf59e16674efd5a53cc (127.21.175.65:41645)
I20260810 03:41:48.594048 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:48.588555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:45987 (local address 127.21.175.67:43485)
0810 03:41:48.588684 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 03:41:48.588705 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:41:48.588818 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:41:48.589537 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:41:48.589540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:41:48.589562 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:41:48.589703 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:41:48.589711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.590901 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.590906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:41:48.592372 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:41:48.592386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.593552 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.593556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:41:48.593680 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:48.593711 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 03:41:48.593800 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 03:41:48.593891 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 03:41:48.595425 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:48.589047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:43485 (local address 127.21.175.66:45987)
0810 03:41:48.589255 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 03:41:48.589260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:41:48.589277 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:41:48.589335 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:41:48.589338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:41:48.589386 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:41:48.589475 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:41:48.589852 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.590740 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.592558 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.593242 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.593379 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:48.593442 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 03:41:48.595187 (+ 1745us) server_negotiation.cc:299] Negotiation successful
0810 03:41:48.595285 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260810 03:41:48.596151 2076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646a97d534284d0686f405316d4da159" candidate_uuid: "a24d26e24e504db5b96f1164e329d1c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d65c0b8752491985a22e544fc97c7d" is_pre_election: true
I20260810 03:41:48.596520 2076 raft_consensus.cc:2454] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a24d26e24e504db5b96f1164e329d1c6 in term 0.
I20260810 03:41:48.597141 2168 leader_election.cc:302] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d65c0b8752491985a22e544fc97c7d, a24d26e24e504db5b96f1164e329d1c6; no voters:
I20260810 03:41:48.597519 2323 raft_consensus.cc:2781] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:41:48.597630 2323 raft_consensus.cc:482] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:41:48.597689 2323 raft_consensus.cc:3037] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:41:48.598791 2323 raft_consensus.cc:504] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d65c0b8752491985a22e544fc97c7d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45987 } } peers { permanent_uuid: "a24d26e24e504db5b96f1164e329d1c6" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46055 } } peers { permanent_uuid: "8ff4cd659f1d4cf59e16674efd5a53cc" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41645 } }
I20260810 03:41:48.599244 2323 leader_election.cc:288] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [CANDIDATE]: Term 1 election: Requested vote from peers 91d65c0b8752491985a22e544fc97c7d (127.21.175.66:45987), 8ff4cd659f1d4cf59e16674efd5a53cc (127.21.175.65:41645)
I20260810 03:41:48.599769 2076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646a97d534284d0686f405316d4da159" candidate_uuid: "a24d26e24e504db5b96f1164e329d1c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d65c0b8752491985a22e544fc97c7d"
I20260810 03:41:48.599900 2076 raft_consensus.cc:3037] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:41:48.600315 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:48.588883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:41645 (local address 127.21.175.67:59237)
0810 03:41:48.594255 (+ 5372us) negotiation.cc:107] Waiting for socket to connect
0810 03:41:48.594273 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:41:48.594370 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:41:48.596027 (+ 1657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:41:48.596031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:41:48.596056 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:41:48.596151 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:41:48.596158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.597539 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.597542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:41:48.598952 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:41:48.598967 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.599844 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.599848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:41:48.599954 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:48.599984 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 03:41:48.600067 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 03:41:48.600143 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5297,"thread_start_us":147,"threads_started":1}
I20260810 03:41:48.600818 2311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:48.589514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:59237 (local address 127.21.175.65:41645)
0810 03:41:48.589963 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0810 03:41:48.589968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:41:48.594712 (+ 4744us) server_negotiation.cc:407] Connection header received
0810 03:41:48.594778 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:41:48.594782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:41:48.594841 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:41:48.594915 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:41:48.596312 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.597190 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.599111 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.599718 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.599855 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:48.599916 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 03:41:48.600594 (+ 678us) server_negotiation.cc:299] Negotiation successful
0810 03:41:48.600685 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360}
I20260810 03:41:48.601230 2076 raft_consensus.cc:2454] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a24d26e24e504db5b96f1164e329d1c6 in term 1.
I20260810 03:41:48.601435 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646a97d534284d0686f405316d4da159" candidate_uuid: "a24d26e24e504db5b96f1164e329d1c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ff4cd659f1d4cf59e16674efd5a53cc" is_pre_election: true
I20260810 03:41:48.601640 2168 leader_election.cc:302] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d65c0b8752491985a22e544fc97c7d, a24d26e24e504db5b96f1164e329d1c6; no voters:
I20260810 03:41:48.601795 1943 raft_consensus.cc:2454] T 646a97d534284d0686f405316d4da159 P 8ff4cd659f1d4cf59e16674efd5a53cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a24d26e24e504db5b96f1164e329d1c6 in term 0.
I20260810 03:41:48.601852 2323 raft_consensus.cc:2781] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:41:48.602164 2323 raft_consensus.cc:686] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [term 1 LEADER]: Becoming Leader. State: Replica: a24d26e24e504db5b96f1164e329d1c6, State: Running, Role: LEADER
I20260810 03:41:48.602259 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646a97d534284d0686f405316d4da159" candidate_uuid: "a24d26e24e504db5b96f1164e329d1c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ff4cd659f1d4cf59e16674efd5a53cc"
I20260810 03:41:48.602378 1942 raft_consensus.cc:3037] T 646a97d534284d0686f405316d4da159 P 8ff4cd659f1d4cf59e16674efd5a53cc [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:41:48.602533 2323 consensus_queue.cc:237] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d65c0b8752491985a22e544fc97c7d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45987 } } peers { permanent_uuid: "a24d26e24e504db5b96f1164e329d1c6" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46055 } } peers { permanent_uuid: "8ff4cd659f1d4cf59e16674efd5a53cc" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41645 } }
I20260810 03:41:48.603844 1942 raft_consensus.cc:2454] T 646a97d534284d0686f405316d4da159 P 8ff4cd659f1d4cf59e16674efd5a53cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a24d26e24e504db5b96f1164e329d1c6 in term 1.
I20260810 03:41:48.606068 1755 catalog_manager.cc:5495] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a24d26e24e504db5b96f1164e329d1c6 (127.21.175.67). New cstate: current_term: 1 leader_uuid: "a24d26e24e504db5b96f1164e329d1c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d65c0b8752491985a22e544fc97c7d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a24d26e24e504db5b96f1164e329d1c6" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ff4cd659f1d4cf59e16674efd5a53cc" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41645 } health_report { overall_health: UNKNOWN } } }
W20260810 03:41:48.633450 1993 tablet.cc:2369] T 646a97d534284d0686f405316d4da159 P 8ff4cd659f1d4cf59e16674efd5a53cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:41:48.673471 2309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:48.669631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.21.175.67:46055)
0810 03:41:48.669774 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 03:41:48.669778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:41:48.669796 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:41:48.669895 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:41:48.669899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:41:48.669975 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:41:48.670112 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:41:48.670464 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.670978 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.671892 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:41:48.672122 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:41:48.672238 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:41:48.672578 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:41:48.673039 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:41:48.673184 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 03:41:48.673221 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 03:41:48.673281 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:41:48.681666 2076 raft_consensus.cc:1261] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d [term 1 FOLLOWER]: Refusing update from remote peer a24d26e24e504db5b96f1164e329d1c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:41:48.681686 1942 raft_consensus.cc:1261] T 646a97d534284d0686f405316d4da159 P 8ff4cd659f1d4cf59e16674efd5a53cc [term 1 FOLLOWER]: Refusing update from remote peer a24d26e24e504db5b96f1164e329d1c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:41:48.682513 2323 consensus_queue.cc:1035] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d65c0b8752491985a22e544fc97c7d" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:41:48.682762 2327 consensus_queue.cc:1035] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ff4cd659f1d4cf59e16674efd5a53cc" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 41645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:41:48.693409 2339 mvcc.cc:205] Tried to move back new op lower bound from 7316821232350900224 to 7316821232039116800. Current Snapshot: MvccSnapshot[applied={T|T < 7316821232350900224}]
I20260810 03:41:48.695802 2340 mvcc.cc:205] Tried to move back new op lower bound from 7316821232350900224 to 7316821232039116800. Current Snapshot: MvccSnapshot[applied={T|T < 7316821232350900224}]
I20260810 03:41:48.708606 1754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:41:48.708838 1754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:41:48.712128 1754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 375788d140f248fb9a00b8463a4ca518: Sending DeleteTablet for 3 replicas of tablet 646a97d534284d0686f405316d4da159
I20260810 03:41:48.713289 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet 646a97d534284d0686f405316d4da159 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:41:48 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:48020
I20260810 03:41:48.714614 2219 tablet_service.cc:1514] Processing DeleteTablet for tablet 646a97d534284d0686f405316d4da159 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:41:48 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:54876
I20260810 03:41:48.714582 1919 tablet_service.cc:1514] Processing DeleteTablet for tablet 646a97d534284d0686f405316d4da159 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:41:48 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:39034
I20260810 03:41:48.715926 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 1845
I20260810 03:41:48.721822 2350 tablet_replica.cc:330] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d: stopping tablet replica
I20260810 03:41:48.722276 2350 raft_consensus.cc:2229] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:41:48.722728 2350 raft_consensus.cc:2258] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:41:48.723367 2352 tablet_replica.cc:330] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6: stopping tablet replica
I20260810 03:41:48.723694 2352 raft_consensus.cc:2229] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:41:48.724133 2352 raft_consensus.cc:2258] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:41:48.726002 2352 ts_tablet_manager.cc:1905] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:41:48.729714 2352 ts_tablet_manager.cc:1918] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:41:48.729822 2352 log.cc:1196] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/646a97d534284d0686f405316d4da159
I20260810 03:41:48.730170 2352 ts_tablet_manager.cc:1939] T 646a97d534284d0686f405316d4da159 P a24d26e24e504db5b96f1164e329d1c6: Deleting consensus metadata
I20260810 03:41:48.731714 1742 catalog_manager.cc:4841] TS a24d26e24e504db5b96f1164e329d1c6 (127.21.175.67:46055): tablet 646a97d534284d0686f405316d4da159 (table test-table [id=022a255f115d4cd2b5ff0ade15e9dc45]) successfully deleted
I20260810 03:41:48.740356 2350 ts_tablet_manager.cc:1905] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:41:48.744120 2350 ts_tablet_manager.cc:1918] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:41:48.744235 2350 log.cc:1196] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/646a97d534284d0686f405316d4da159
I20260810 03:41:48.744612 2350 ts_tablet_manager.cc:1939] T 646a97d534284d0686f405316d4da159 P 91d65c0b8752491985a22e544fc97c7d: Deleting consensus metadata
I20260810 03:41:48.750926 1739 catalog_manager.cc:4841] TS 91d65c0b8752491985a22e544fc97c7d (127.21.175.66:45987): tablet 646a97d534284d0686f405316d4da159 (table test-table [id=022a255f115d4cd2b5ff0ade15e9dc45]) successfully deleted
W20260810 03:41:48.751570 1742 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.175.65:41645 (error 108)
I20260810 03:41:48.752744 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 1998
I20260810 03:41:48.755512 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:41:48.755121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:41645 (local address 127.0.0.1:39042)
0810 03:41:48.755328 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 03:41:48.755402 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:41645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260810 03:41:48.755831 1742 catalog_manager.cc:4584] TS 8ff4cd659f1d4cf59e16674efd5a53cc (127.21.175.65:41645): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 646a97d534284d0686f405316d4da159: Network error: Client connection negotiation failed: client connection to 127.21.175.65:41645: connect: Connection refused (error 111)
I20260810 03:41:48.764338 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 2151
I20260810 03:41:48.773860 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 1723
I20260810 03:41:48.783236 22205 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 03:41:49.129069 22205 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 03:41:49.129244 22205 mini_ranger.cc:66] Stopping Ranger...
I20260810 03:41:49.475670 22205 mini_ranger.cc:68] Stopped Ranger
2026-08-10 03:41:49.475 UTC [1273] LOG: received smart shutdown request
2026-08-10 03:41:49.480 UTC [1273] LOG: background worker "logical replication launcher" (PID 1284) exited with exit code 1
2026-08-10 03:41:49.480 UTC [1278] LOG: shutting down
2026-08-10 03:41:49.699 UTC [1273] LOG: database system is shut down
2026-08-10T03:41:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50656 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:41:49 dist-test-slave-ss4n krb5kdc[2365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:41:49 dist-test-slave-ss4n krb5kdc[2365](info): set up 2 sockets
Aug 10 03:41:49 dist-test-slave-ss4n krb5kdc[2365](info): commencing operation
krb5kdc: starting...
W20260810 03:41:50.948164 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:41:51 dist-test-slave-ss4n krb5kdc[2365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:41:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:41:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.175.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.175.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.175.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/rangeradmin_127.21.175.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.175.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/rangeradmin_127.21.175.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.175.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.175.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.175.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/rangerlookup_127.21.175.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.175.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/rangerlookup_127.21.175.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.175.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.175.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.175.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/HTTP_127.21.175.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.175.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/HTTP_127.21.175.84@KRBTEST.COM.keytab.
I20260810 03:41:51.149112 22205 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMmCxq6/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/postgres -l logfile start
2026-08-10 03:41:53.985 UTC [2396] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 03:41:53.985 UTC [2396] LOG: listening on IPv4 address "127.21.175.84", port 40597
2026-08-10 03:41:53.989 UTC [2396] LOG: listening on Unix socket "/tmp/.s.PGSQL.40597"
2026-08-10 03:41:54.024 UTC [2399] LOG: database system was shut down at 2026-08-10 03:41:52 UTC
2026-08-10 03:41:54.028 UTC [2396] LOG: database system is ready to accept connections
I20260810 03:41:55.084299 22205 mini_postgres.cc:96] Postgres bound to 40597
2026-08-10 03:41:55.092 UTC [2407] FATAL: database "slave" does not exist
127.21.175.84:40597 - accepting connections
I20260810 03:41:55.093070 22205 mini_ranger.cc:161] Starting Ranger...
I20260810 03:41:55.108902 22205 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 03:41:55.671334 22205 mini_ranger.cc:87] Created ranger Postgres database
I20260810 03:41:55.671437 22205 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-admin
2026-08-10 03:41:55,967 [I] DB FLAVOR :POSTGRES
2026-08-10 03:41:55,967 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 03:41:55,967 [I] Checking connection..
2026-08-10 03:41:55,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 03:41:56,242 [I] Checking connection passed.
2026-08-10 03:41:56,242 [I] --------- Verifying version history table ---------
2026-08-10 03:41:56,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:41:56,504 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 03:41:56,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:41:56,794 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 03:41:56,794 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 03:41:56,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 03:41:57.060 UTC [2477] WARNING: there is no transaction in progress
2026-08-10 03:41:57,068 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 03:41:57,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:41:57,313 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 03:41:57,314 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 03:41:57,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 03:41:57,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 03:41:57,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ss4n.c.gcp-upstream.internal','N') ;"
2026-08-10 03:41:58,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 03:41:58,430 [I] Table x_portal_user does not exist in database ranger
2026-08-10 03:41:58,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 03:41:58,700 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 03:41:58,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 03:41:58,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 03:41:59,227 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 03:41:59,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 03:42:00.304 UTC [2594] WARNING: there is no transaction in progress
2026-08-10 03:42:00.323 UTC [2594] WARNING: there is no transaction in progress
2026-08-10 03:42:00.343 UTC [2594] WARNING: there is no transaction in progress
2026-08-10 03:42:00.361 UTC [2594] WARNING: there is no transaction in progress
2026-08-10 03:42:00.380 UTC [2594] WARNING: there is no transaction in progress
2026-08-10 03:42:00.401 UTC [2594] WARNING: there is no transaction in progress
2026-08-10 03:42:00.461 UTC [2594] WARNING: there is no transaction in progress
2026-08-10 03:42:00.470 UTC [2594] WARNING: there is no transaction in progress
2026-08-10 03:42:00.486 UTC [2594] WARNING: there is no transaction in progress
2026-08-10 03:42:00.496 UTC [2594] WARNING: there is no transaction in progress
2026-08-10 03:42:00.507 UTC [2594] WARNING: there is no transaction in progress
2026-08-10 03:42:00.522 UTC [2594] WARNING: there is no transaction in progress
2026-08-10 03:42:00.534 UTC [2594] WARNING: there is no transaction in progress
2026-08-10 03:42:00.542 UTC [2594] WARNING: there is no transaction in progress
2026-08-10 03:42:00.550 UTC [2594] WARNING: there is no transaction in progress
2026-08-10 03:42:00.559 UTC [2594] WARNING: there is no transaction in progress
2026-08-10 03:42:01,303 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 03:42:01,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 03:42:01,561 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 03:42:01,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 03:42:01,837 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 03:42:01,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 03:42:02,115 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 03:42:02,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 03:42:02,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 03:42:02,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ss4n.c.gcp-upstream.internal';"
2026-08-10 03:42:02,897 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 03:42:02,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 03:42:03,174 [I] DB_PATCHES have already been applied
I20260810 03:42:03.180554 22205 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMmCxq6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 03:42:03.180629 22205 mini_ranger.cc:193] Using host: 127.21.175.84
I20260810 03:42:03.184965 22205 mini_ranger.cc:227] Ranger admin URL: http://127.21.175.84:60805
Aug 10, 2026 3:42:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:42:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:42:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 3:42:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 3:42:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 3:42:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 3:42:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 03:42:04.417205 22205 mini_ranger.cc:160] Time spent starting Ranger: real 9.324s user 0.000s sys 0.012s
Aug 10, 2026 3:42:25 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 3:42:25 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 3:42:25 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 3:42:25 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 3:42:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 3:42:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 3:42:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 3:42:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 3:42:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 3:42:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 3:42:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 3:42:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 3:42:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 3:42:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 3:42:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 3:42:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 3:42:25 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 3:42:26 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 03:42:27.075495 22205 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.175.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.175.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.175.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/rangerkms_127.21.175.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.175.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/rangerkms_127.21.175.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.175.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.175.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.175.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/HTTP_127.21.175.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.175.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/HTTP_127.21.175.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 03:42:27 dist-test-slave-ss4n krb5kdc[2365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333347, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 03:42:27.173115 22205 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 03:42:27.184291 22205 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 03:42:28.171341 22205 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 03:42:28.171593 22205 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-kms
I20260810 03:42:28.171631 22205 mini_ranger_kms.cc:227] Using postgres at 127.21.175.84:40597
2026-08-10 03:42:28,355 [I] DB FLAVOR :POSTGRES
2026-08-10 03:42:28,356 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 03:42:28,356 [I] Checking connection
2026-08-10 03:42:28,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 03:42:28,602 [I] connection success
2026-08-10 03:42:28,603 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 03:42:28,603 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 03:42:28,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 03:42:28,845 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 03:42:28,845 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 03:42:28,845 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 03:42:28,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.175.84:40597/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 03:42:29,131 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 03:42:29.634521 22205 mini_ranger_kms.cc:310] Created kms service
I20260810 03:42:29.872274 22205 mini_ranger_kms.cc:326] Created kudu user
I20260810 03:42:29.922089 22205 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 03:42:29 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333347, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
I20260810 03:42:30.369616 22205 mini_ranger_kms.cc:384] Added ranger policy
I20260810 03:42:30.369784 22205 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMmCxq6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMmCxq6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 03:42:30.369807 22205 mini_ranger_kms.cc:240] Using host: 127.21.175.84
I20260810 03:42:30.375075 22205 mini_ranger_kms.cc:276] Ranger KMS PID: 2790
I20260810 03:42:30.375177 22205 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.175.84:49917
Aug 10, 2026 3:42:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:42:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:42:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 3:42:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 3:42:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 3:42:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 03:42:32 dist-test-slave-ss4n krb5kdc[2365](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786333352, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.175.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:32.839743 22205 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.667s user 0.013s sys 0.156s
I20260810 03:42:32.839876 22205 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 03:42:32.839951 22205 mini_ranger_kms.cc:401] 127.21.175.84:49917/kms/v1/keys
Aug 10 03:42:33 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333352, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.175.84@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
Aug 10 03:42:33 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333352, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.175.84@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
Aug 10 03:42:35 dist-test-slave-ss4n krb5kdc[2365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:42:35.942596 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:39945
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:39945
--ranger_config_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.175.84:49917/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:42:36.080902 2846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:36.081202 2846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:36.081243 2846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:36.085408 2846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:42:36.085546 2846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:36.085587 2846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:36.085644 2846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:42:36.085665 2846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:42:36.091496 2846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.175.84:49917/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:39945
--ranger_config_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:39945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:36.092674 2846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:36.093855 2846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:36.102911 2851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:36.102926 2852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:36.103363 2854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:36.103446 2846 server_base.cc:1034] running on GCE node
I20260810 03:42:36.104049 2846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:36.105346 2846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:36.106591 2846 hybrid_clock.cc:648] HybridClock initialized: now 1786333356106568 us; error 42 us; skew 500 ppm
Aug 10 03:42:36 dist-test-slave-ss4n krb5kdc[2365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333356, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:36.115244 2846 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:42:36.116359 2846 webserver.cc:460] Webserver started at http://127.21.175.126:33795/ using document root <none> and password file <none>
I20260810 03:42:36.116997 2846 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:36.117085 2846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:36.117341 2846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:42:36 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333356, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
I20260810 03:42:36.172932 2846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "26eff2c679fb4fc1a483881229f47be8"
format_stamp: "Formatted at 2026-08-10 03:42:36 on dist-test-slave-ss4n"
server_key: "3e809049fb029fadb83ec354eaf8c62e"
server_key_iv: "f37875a718585f3dd469b8925e1dd129"
server_key_version: "kuduclusterkey@0"
I20260810 03:42:36.173705 2846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "26eff2c679fb4fc1a483881229f47be8"
format_stamp: "Formatted at 2026-08-10 03:42:36 on dist-test-slave-ss4n"
server_key: "3e809049fb029fadb83ec354eaf8c62e"
server_key_iv: "f37875a718585f3dd469b8925e1dd129"
server_key_version: "kuduclusterkey@0"
I20260810 03:42:36.178009 2846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:42:36.197182 2861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:36.198860 2846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:42:36.199028 2846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "26eff2c679fb4fc1a483881229f47be8"
format_stamp: "Formatted at 2026-08-10 03:42:36 on dist-test-slave-ss4n"
server_key: "3e809049fb029fadb83ec354eaf8c62e"
server_key_iv: "f37875a718585f3dd469b8925e1dd129"
server_key_version: "kuduclusterkey@0"
I20260810 03:42:36.199182 2846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:36.235260 2846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:36.238541 2846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:36.238762 2846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:36.247362 2846 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:39945
I20260810 03:42:36.247366 2918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:39945 every 8 connection(s)
I20260810 03:42:36.248452 2846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:42:36.251475 2919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:42:36.253393 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 2846
I20260810 03:42:36.253545 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:42:36.253885 22205 external_mini_cluster.cc:1428] Setting key 14aaba63d128b5879214e97ec0d2ec04
I20260810 03:42:36.258885 2919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8: Bootstrap starting.
I20260810 03:42:36.261528 2919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8: Neither blocks nor log segments found. Creating new log.
I20260810 03:42:36.262482 2919 log.cc:824] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:42:36 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:42:36.265367 2919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8: No bootstrap required, opened a new log
I20260810 03:42:36.270300 2923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:36.255446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54464 (local address 127.21.175.126:39945)
0810 03:42:36.255964 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:36.255985 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:36.256032 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:42:36.256953 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:36.256980 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:36.257422 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:36.257859 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:36.258782 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:36.260329 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:36.261615 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:36.262382 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:36.262560 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:36.265394 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:36.265399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:36.265414 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:36.265471 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:36.267242 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:36.267728 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:36.267733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:36.267738 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:36.267940 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:36.268249 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:36.268254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:36.268256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:36.268678 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:36.268885 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:36.269197 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 03:42:36.269546 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":170,"threads_started":1}
I20260810 03:42:36.270365 2919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26eff2c679fb4fc1a483881229f47be8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39945 } }
I20260810 03:42:36.270705 2919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:42:36.270792 2919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26eff2c679fb4fc1a483881229f47be8, State: Initialized, Role: FOLLOWER
I20260810 03:42:36.271342 2919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26eff2c679fb4fc1a483881229f47be8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39945 } }
I20260810 03:42:36.271628 2919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:42:36.271719 2919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:42:36.271867 2919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:42:36.273182 2919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26eff2c679fb4fc1a483881229f47be8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39945 } }
I20260810 03:42:36.273597 2919 leader_election.cc:302] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26eff2c679fb4fc1a483881229f47be8; no voters:
I20260810 03:42:36.273991 2919 leader_election.cc:288] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:42:36.274127 2925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:42:36.274377 2925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [term 1 LEADER]: Becoming Leader. State: Replica: 26eff2c679fb4fc1a483881229f47be8, State: Running, Role: LEADER
I20260810 03:42:36.274772 2925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26eff2c679fb4fc1a483881229f47be8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39945 } }
I20260810 03:42:36.275384 2919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:42:36.276036 2925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26eff2c679fb4fc1a483881229f47be8. Latest consensus state: current_term: 1 leader_uuid: "26eff2c679fb4fc1a483881229f47be8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26eff2c679fb4fc1a483881229f47be8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39945 } } }
I20260810 03:42:36.276741 2927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26eff2c679fb4fc1a483881229f47be8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26eff2c679fb4fc1a483881229f47be8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39945 } } }
I20260810 03:42:36.276971 2927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:42:36.276971 2925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:42:36.277874 2919 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 03:42:36.279278 2936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:42:36.286197 2936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:42:36.294137 2936 catalog_manager.cc:1347] Generated new cluster ID: 7826c1ad4fa64df78be126d6ede778b3
I20260810 03:42:36.294219 2936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:42:36.307309 2936 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:42:36.308724 2936 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:42:36.321262 2936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8: Generated new TSK 0
I20260810 03:42:36.322482 2936 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:42:37.108557 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.175.84:49917/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39945
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:42:37.258095 2968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:37.258533 2968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:37.258662 2968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:37.262964 2968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:37.263053 2968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:37.263152 2968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:42:37.269430 2968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.175.84:49917/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:37.270844 2968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:37.271965 2968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:37.284765 2979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:37.284818 2977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:37.284797 2982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:37.286182 2968 server_base.cc:1034] running on GCE node
I20260810 03:42:37.286625 2968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:37.287482 2968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:37.288712 2968 hybrid_clock.cc:648] HybridClock initialized: now 1786333357288675 us; error 63 us; skew 500 ppm
Aug 10 03:42:37 dist-test-slave-ss4n krb5kdc[2365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333357, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:37.291836 2968 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:42:37.292896 2968 webserver.cc:460] Webserver started at http://127.21.175.65:41019/ using document root <none> and password file <none>
I20260810 03:42:37.293583 2968 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:37.293684 2968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:37.293910 2968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:42:37 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333357, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
I20260810 03:42:37.317636 2968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "ceda524dc1ca434cac517f9d49fc6678"
format_stamp: "Formatted at 2026-08-10 03:42:37 on dist-test-slave-ss4n"
server_key: "5c7ab1eeeff74c20e5ac9ffcbcddf59c"
server_key_iv: "7f94fcebfc568d842b1a86f340e334ad"
server_key_version: "kuduclusterkey@0"
I20260810 03:42:37.318712 2968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "ceda524dc1ca434cac517f9d49fc6678"
format_stamp: "Formatted at 2026-08-10 03:42:37 on dist-test-slave-ss4n"
server_key: "5c7ab1eeeff74c20e5ac9ffcbcddf59c"
server_key_iv: "7f94fcebfc568d842b1a86f340e334ad"
server_key_version: "kuduclusterkey@0"
I20260810 03:42:37.324502 2968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:42:37.352790 2989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:37.354789 2968 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260810 03:42:37.354952 2968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "ceda524dc1ca434cac517f9d49fc6678"
format_stamp: "Formatted at 2026-08-10 03:42:37 on dist-test-slave-ss4n"
server_key: "5c7ab1eeeff74c20e5ac9ffcbcddf59c"
server_key_iv: "7f94fcebfc568d842b1a86f340e334ad"
server_key_version: "kuduclusterkey@0"
I20260810 03:42:37.355122 2968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:37.381618 2968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:37.385878 2968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:37.386178 2968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:37.386967 2968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:42:37.388203 2968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:42:37.388330 2968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:37.388484 2968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:42:37.388576 2968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:37.400496 2968 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:42217
I20260810 03:42:37.402339 2968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:42:37.412607 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 2968
I20260810 03:42:37.412744 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:42:37.413090 22205 external_mini_cluster.cc:1428] Setting key 76509bc4c5dd660acf86b5d696f7dfb6
I20260810 03:42:37.413708 3112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:42217 every 8 connection(s)
Aug 10 03:42:37 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333357, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
I20260810 03:42:37.435570 3115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:37.416451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:39945 (local address 127.21.175.65:45773)
0810 03:42:37.417969 (+ 1518us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:37.418035 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:37.419223 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:37.420073 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:37.420084 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:37.420501 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:37.421291 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:37.421315 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:37.422691 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:37.422697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:37.424337 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:37.424351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:37.425250 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:37.425255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:37.425544 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:37.426399 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:37.426423 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:37.428492 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:37.431495 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:37.431505 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:37.431527 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:37.431966 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:37.432408 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:37.432412 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:37.432414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:37.432644 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:37.433213 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:37.433222 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:37.433673 (+ 451us) client_negotiation.cc:769] Sending connection context
0810 03:42:37.433947 (+ 274us) client_negotiation.cc:240] Negotiation successful
0810 03:42:37.434339 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1242,"thread_start_us":188,"threads_started":1}
I20260810 03:42:37.436465 3118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:37.417715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:45773 (local address 127.21.175.126:39945)
0810 03:42:37.418003 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:37.418008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:37.418163 (+ 155us) server_negotiation.cc:407] Connection header received
0810 03:42:37.419515 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:37.419520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:37.419597 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:37.419711 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:37.421527 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:37.422504 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:37.424521 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:37.424914 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:37.425034 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:37.428723 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:37.428727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:37.428731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:37.428762 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:37.431309 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:37.432115 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:37.432119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:37.432122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:37.432280 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:37.432774 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:37.432779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:37.432781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:37.433045 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:37.433150 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:37.436073 (+ 2923us) server_negotiation.cc:299] Negotiation successful
0810 03:42:37.436260 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":90,"threads_started":1}
I20260810 03:42:37.437893 3113 heartbeater.cc:347] Connected to a master server at 127.21.175.126:39945
I20260810 03:42:37.438186 3113 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:37.438827 3113 heartbeater.cc:511] Master 127.21.175.126:39945 requested a full tablet report, sending...
I20260810 03:42:37.440809 2878 ts_manager.cc:194] Registered new tserver with Master: ceda524dc1ca434cac517f9d49fc6678 (127.21.175.65:42217)
I20260810 03:42:37.442551 2878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:45773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:42:37.457370 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.175.84:49917/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39945
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:42:37 dist-test-slave-ss4n krb5kdc[2365](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786333357, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 03:42:37.634227 3121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:37.634771 3121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:37.634934 3121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:37.642057 3121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:37.642179 3121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:37.642297 3121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:42:37.651540 3121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.175.84:49917/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:37.653343 3121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:37.654959 3121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:37.669420 3129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:37.669597 3131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:37.669395 3128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:37.671128 3121 server_base.cc:1034] running on GCE node
I20260810 03:42:37.671695 3121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:37.672645 3121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:37.675961 3121 hybrid_clock.cc:648] HybridClock initialized: now 1786333357675836 us; error 131 us; skew 500 ppm
Aug 10 03:42:37 dist-test-slave-ss4n krb5kdc[2365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333357, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:37.680289 3121 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:42:37.681747 3121 webserver.cc:460] Webserver started at http://127.21.175.66:42401/ using document root <none> and password file <none>
I20260810 03:42:37.682521 3121 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:37.682622 3121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:37.682878 3121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:42:37 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333357, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
I20260810 03:42:37.703684 3121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "c032afb1882b4f428a9be96172c86403"
format_stamp: "Formatted at 2026-08-10 03:42:37 on dist-test-slave-ss4n"
server_key: "21db8997a34c019ae0fbb4c07b0a1fb1"
server_key_iv: "8d61cfc6572669fe65548362b6295acb"
server_key_version: "kuduclusterkey@0"
I20260810 03:42:37.704682 3121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "c032afb1882b4f428a9be96172c86403"
format_stamp: "Formatted at 2026-08-10 03:42:37 on dist-test-slave-ss4n"
server_key: "21db8997a34c019ae0fbb4c07b0a1fb1"
server_key_iv: "8d61cfc6572669fe65548362b6295acb"
server_key_version: "kuduclusterkey@0"
I20260810 03:42:37.710836 3121 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260810 03:42:37.734697 3140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:37.736791 3121 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s
I20260810 03:42:37.736958 3121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "c032afb1882b4f428a9be96172c86403"
format_stamp: "Formatted at 2026-08-10 03:42:37 on dist-test-slave-ss4n"
server_key: "21db8997a34c019ae0fbb4c07b0a1fb1"
server_key_iv: "8d61cfc6572669fe65548362b6295acb"
server_key_version: "kuduclusterkey@0"
I20260810 03:42:37.737116 3121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:37.766317 3121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:37.769809 3121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:37.770036 3121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:37.770705 3121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:42:37.771646 3121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:42:37.771716 3121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:37.771814 3121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:42:37.771862 3121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:37.783083 3121 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:40333
I20260810 03:42:37.783208 3265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:40333 every 8 connection(s)
I20260810 03:42:37.784430 3121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:42:37.785144 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 3121
I20260810 03:42:37.785293 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:42:37.785679 22205 external_mini_cluster.cc:1428] Setting key 0bf1a3bd89662bb0cad19eea5120359b
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
Aug 10 03:42:37 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333357, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
I20260810 03:42:37.809763 3118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:37.789922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:51959 (local address 127.21.175.126:39945)
0810 03:42:37.790076 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:37.790082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:37.790569 (+ 487us) server_negotiation.cc:407] Connection header received
0810 03:42:37.791846 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:37.791850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:37.791906 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:37.791986 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:37.793712 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:37.794615 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:37.796596 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:37.796987 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:37.797101 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:37.802861 (+ 5760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:37.802865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:37.802868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:37.802902 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:37.807023 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:37.807817 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:37.807822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:37.807824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:37.807918 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:37.808325 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:37.808329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:37.808331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:37.808588 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:37.808657 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:37.809416 (+ 759us) server_negotiation.cc:299] Negotiation successful
0810 03:42:37.809590 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:42:37.810771 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:37.788386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:39945 (local address 127.21.175.66:51959)
0810 03:42:37.790389 (+ 2003us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:37.790455 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:37.791630 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:37.792324 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:37.792337 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:37.792722 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:37.793468 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:37.793540 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:37.794797 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:37.794801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:37.796432 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:37.796447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:37.797352 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:37.797356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:37.797621 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:37.798412 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:37.798442 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:37.802641 (+ 4199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:37.807221 (+ 4580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:37.807229 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:37.807253 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:37.807686 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:37.808026 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:37.808029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:37.808031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:37.808223 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:37.808752 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:37.808759 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:37.809140 (+ 381us) client_negotiation.cc:769] Sending connection context
0810 03:42:37.809358 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 03:42:37.809781 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1747,"thread_start_us":154,"threads_started":1}
I20260810 03:42:37.812059 3266 heartbeater.cc:347] Connected to a master server at 127.21.175.126:39945
I20260810 03:42:37.812335 3266 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:37.812955 3266 heartbeater.cc:511] Master 127.21.175.126:39945 requested a full tablet report, sending...
I20260810 03:42:37.814198 2878 ts_manager.cc:194] Registered new tserver with Master: c032afb1882b4f428a9be96172c86403 (127.21.175.66:40333)
I20260810 03:42:37.815044 2878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:51959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:42:37.833596 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.175.84:49917/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39945
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:42:38.002364 3272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:38.002815 3272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:38.002940 3272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:38.010509 3272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:38.010687 3272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:38.010854 3272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:42:38.021140 3272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.175.84:49917/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:39945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:38.023299 3272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:38.024809 3272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:38.039441 3279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:38.039870 3272 server_base.cc:1034] running on GCE node
W20260810 03:42:38.041658 3278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:38.042040 3281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:38.042659 3272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:38.043567 3272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:38.046828 3272 hybrid_clock.cc:648] HybridClock initialized: now 1786333358046732 us; error 108 us; skew 500 ppm
Aug 10 03:42:38 dist-test-slave-ss4n krb5kdc[2365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333358, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 03:42:38 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333357, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
I20260810 03:42:38.051152 3272 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:42:38.052229 3272 webserver.cc:460] Webserver started at http://127.21.175.67:35979/ using document root <none> and password file <none>
I20260810 03:42:38.053009 3272 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:38.053120 3272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:38.053388 3272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:42:38 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333358, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
I20260810 03:42:38.102509 3272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "a9740a0313924d839ded6b6e7dde6e89"
format_stamp: "Formatted at 2026-08-10 03:42:38 on dist-test-slave-ss4n"
server_key: "2ce104b628e9a95bdec7e466323b27c3"
server_key_iv: "77684ca5586f72319d4e4b08ed185719"
server_key_version: "kuduclusterkey@0"
I20260810 03:42:38.103420 3272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "a9740a0313924d839ded6b6e7dde6e89"
format_stamp: "Formatted at 2026-08-10 03:42:38 on dist-test-slave-ss4n"
server_key: "2ce104b628e9a95bdec7e466323b27c3"
server_key_iv: "77684ca5586f72319d4e4b08ed185719"
server_key_version: "kuduclusterkey@0"
I20260810 03:42:38.113749 3272 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.000s
I20260810 03:42:38.158037 3289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:38.160176 3272 fs_manager.cc:725] Time spent opening block manager: real 0.024s user 0.004s sys 0.000s
I20260810 03:42:38.160411 3272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "a9740a0313924d839ded6b6e7dde6e89"
format_stamp: "Formatted at 2026-08-10 03:42:38 on dist-test-slave-ss4n"
server_key: "2ce104b628e9a95bdec7e466323b27c3"
server_key_iv: "77684ca5586f72319d4e4b08ed185719"
server_key_version: "kuduclusterkey@0"
I20260810 03:42:38.160663 3272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:38.184929 3272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:38.193173 3272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:38.193536 3272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:38.194365 3272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:42:38.195467 3272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:42:38.195564 3272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:38.195644 3272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:42:38.195683 3272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:38.219360 3272 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:35747
I20260810 03:42:38.231081 3272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:42:38.232935 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 3272
I20260810 03:42:38.233069 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:42:38.233398 22205 external_mini_cluster.cc:1428] Setting key 06cb2e9c02c38371f4edce4c18110de9
I20260810 03:42:38.233891 3414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:35747 every 8 connection(s)
Aug 10 03:42:38 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333358, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:42:38.281742 3118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:38.250732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:42323 (local address 127.21.175.126:39945)
0810 03:42:38.250900 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:38.250905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:38.260575 (+ 9670us) server_negotiation.cc:407] Connection header received
0810 03:42:38.262143 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:38.262171 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:38.262232 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:38.262289 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:38.264237 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:38.265123 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:38.272385 (+ 7262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:38.272956 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:38.273067 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:38.276905 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:38.276911 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:38.276914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:38.276948 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:38.278983 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:38.279686 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:38.279690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:38.279692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:38.279791 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:38.280266 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:38.280269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:38.280271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:38.280515 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:38.280598 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:38.281372 (+ 774us) server_negotiation.cc:299] Negotiation successful
0810 03:42:38.281553 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 03:42:38.283795 3417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:38.249462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:39945 (local address 127.21.175.67:42323)
0810 03:42:38.260353 (+ 10891us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:38.260458 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:38.261801 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:38.262812 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:38.262824 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:38.263244 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:38.264047 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:38.264071 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:38.265307 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:38.265311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:38.272218 (+ 6907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:38.272234 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:38.273391 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:38.273398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:38.273725 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:38.274658 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:38.274684 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:38.276668 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:38.279146 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:38.279154 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:38.279174 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:38.279555 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:38.279911 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:38.279914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:38.279916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:38.280144 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:38.280670 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:38.280677 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:38.281049 (+ 372us) client_negotiation.cc:769] Sending connection context
0810 03:42:38.282231 (+ 1182us) client_negotiation.cc:240] Negotiation successful
0810 03:42:38.282633 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10538,"thread_start_us":137,"threads_started":1}
I20260810 03:42:38.285727 3415 heartbeater.cc:347] Connected to a master server at 127.21.175.126:39945
I20260810 03:42:38.286023 3415 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:38.286664 3415 heartbeater.cc:511] Master 127.21.175.126:39945 requested a full tablet report, sending...
I20260810 03:42:38.287892 2878 ts_manager.cc:194] Registered new tserver with Master: a9740a0313924d839ded6b6e7dde6e89 (127.21.175.67:35747)
I20260810 03:42:38.288717 2878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:42323
I20260810 03:42:38.295667 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:42:38.296509 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 2968
I20260810 03:42:38.317338 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 3121
I20260810 03:42:38.331519 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 3272
I20260810 03:42:38.350327 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 2846
I20260810 03:42:38.363076 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:39945
--webserver_interface=127.21.175.126
--webserver_port=33795
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:39945
--ranger_config_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.175.84:49917/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.175.84:49917/kms with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:42:38.478669 3418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:38.478974 3418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:38.479014 3418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:38.483160 3418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:42:38.483228 3418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:38.483247 3418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:38.483264 3418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:42:38.483320 3418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:42:38.488997 3418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.175.84:49917/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:39945
--ranger_config_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:39945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=33795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:38.490190 3418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:38.491305 3418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:38.500310 3424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:38.500348 3426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:38.500310 3423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:38.500792 3418 server_base.cc:1034] running on GCE node
I20260810 03:42:38.501472 3418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:38.502781 3418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:38.503973 3418 hybrid_clock.cc:648] HybridClock initialized: now 1786333358503946 us; error 38 us; skew 500 ppm
Aug 10 03:42:38 dist-test-slave-ss4n krb5kdc[2365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333358, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:38.507052 3418 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:42:38.508016 3418 webserver.cc:460] Webserver started at http://127.21.175.126:33795/ using document root <none> and password file <none>
I20260810 03:42:38.508622 3418 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:38.508726 3418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 03:42:38 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333358, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
I20260810 03:42:38.592645 3418 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:42:38.595269 3434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:38.596705 3418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:42:38.596820 3418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "26eff2c679fb4fc1a483881229f47be8"
format_stamp: "Formatted at 2026-08-10 03:42:36 on dist-test-slave-ss4n"
server_key: "3e809049fb029fadb83ec354eaf8c62e"
server_key_iv: "f37875a718585f3dd469b8925e1dd129"
server_key_version: "kuduclusterkey@0"
I20260810 03:42:38.597452 3418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:38.627990 3418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:38.631246 3418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:38.631456 3418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:38.639315 3418 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:39945
I20260810 03:42:38.639322 3490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:39945 every 8 connection(s)
I20260810 03:42:38.640468 3418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:42:38.642910 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 3418
I20260810 03:42:38.643410 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.175.84:49917/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:42217
--local_ip_for_outbound_sockets=127.21.175.65
--tserver_master_addrs=127.21.175.126:39945
--webserver_port=41019
--webserver_interface=127.21.175.65
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.175.84:49917/kms with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:42:38.645141 3491 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:42:38.646903 3491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8: Bootstrap starting.
I20260810 03:42:38.665464 3491 log.cc:824] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:42:38.672748 3491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:42:38.673218 3491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8: Bootstrap complete.
I20260810 03:42:38.677856 3491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26eff2c679fb4fc1a483881229f47be8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39945 } }
I20260810 03:42:38.678318 3491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26eff2c679fb4fc1a483881229f47be8, State: Initialized, Role: FOLLOWER
I20260810 03:42:38.678864 3491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26eff2c679fb4fc1a483881229f47be8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39945 } }
I20260810 03:42:38.679031 3491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:42:38.679131 3491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:42:38.679262 3491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:42:38.681329 3491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26eff2c679fb4fc1a483881229f47be8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39945 } }
I20260810 03:42:38.681741 3491 leader_election.cc:302] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26eff2c679fb4fc1a483881229f47be8; no voters:
I20260810 03:42:38.682169 3491 leader_election.cc:288] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 03:42:38.682263 3495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:42:38.682523 3495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [term 2 LEADER]: Becoming Leader. State: Replica: 26eff2c679fb4fc1a483881229f47be8, State: Running, Role: LEADER
I20260810 03:42:38.682921 3495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26eff2c679fb4fc1a483881229f47be8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39945 } }
I20260810 03:42:38.683665 3491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:42:38.684477 3498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "26eff2c679fb4fc1a483881229f47be8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26eff2c679fb4fc1a483881229f47be8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39945 } } }
I20260810 03:42:38.684613 3498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:42:38.684535 3497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26eff2c679fb4fc1a483881229f47be8. Latest consensus state: current_term: 2 leader_uuid: "26eff2c679fb4fc1a483881229f47be8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26eff2c679fb4fc1a483881229f47be8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39945 } } }
I20260810 03:42:38.684666 3497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:42:38.686189 3504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:42:38.686595 3491 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 03:42:38.693708 3504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:42:38.694572 3504 catalog_manager.cc:1259] Loaded cluster ID: 7826c1ad4fa64df78be126d6ede778b3
I20260810 03:42:38.694674 3504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:42:38.697643 3504 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:42:38.698422 3504 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8: Loaded TSK: 0
I20260810 03:42:38.700018 3504 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:42:38.794653 3493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:38.795087 3493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:38.795192 3493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:38.801331 3493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:38.801414 3493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:38.801533 3493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:42:38.809867 3493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.175.84:49917/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:42217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=41019
--tserver_master_addrs=127.21.175.126:39945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:38.811309 3493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:38.812645 3493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:38.825645 3538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:38.825604 3536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:38.825783 3493 server_base.cc:1034] running on GCE node
W20260810 03:42:38.825978 3540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:38.826575 3493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:38.827410 3493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:38.828722 3493 hybrid_clock.cc:648] HybridClock initialized: now 1786333358828683 us; error 54 us; skew 500 ppm
Aug 10 03:42:38 dist-test-slave-ss4n krb5kdc[2365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333358, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:38.832643 3493 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:42:38.833973 3493 webserver.cc:460] Webserver started at http://127.21.175.65:41019/ using document root <none> and password file <none>
I20260810 03:42:38.834810 3493 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:38.835008 3493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 03:42:38 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333358, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
I20260810 03:42:38.860330 3493 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:42:38.869009 3548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:38.874166 3493 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20260810 03:42:38.874411 3493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "ceda524dc1ca434cac517f9d49fc6678"
format_stamp: "Formatted at 2026-08-10 03:42:37 on dist-test-slave-ss4n"
server_key: "5c7ab1eeeff74c20e5ac9ffcbcddf59c"
server_key_iv: "7f94fcebfc568d842b1a86f340e334ad"
server_key_version: "kuduclusterkey@0"
I20260810 03:42:38.875391 3493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:38.895602 3493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:38.900012 3493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:38.900316 3493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:38.901295 3493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:42:38.902778 3493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:42:38.902900 3493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:38.903070 3493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:42:38.903193 3493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:38.915881 3493 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:42217
I20260810 03:42:38.917330 3493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:42:38.917299 3674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:42217 every 8 connection(s)
I20260810 03:42:38.923552 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 3493
I20260810 03:42:38.924309 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.175.84:49917/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:40333
--local_ip_for_outbound_sockets=127.21.175.66
--tserver_master_addrs=127.21.175.126:39945
--webserver_port=42401
--webserver_interface=127.21.175.66
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.175.84:49917/kms with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:42:38 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333358, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:42:38.951889 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:38.920424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:39945 (local address 127.21.175.65:50853)
0810 03:42:38.921268 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:38.921326 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:38.922302 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:38.925981 (+ 3679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:38.925994 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:38.926423 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:38.927149 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:38.927175 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:38.929355 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:38.929361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:38.930944 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:38.930955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:38.937953 (+ 6998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:38.937961 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:38.938274 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:38.939185 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:38.939210 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:38.941217 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:38.947504 (+ 6287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:38.947514 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:38.947535 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:38.948005 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:38.948520 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:38.948524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:38.948526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:38.948724 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:38.949667 (+ 943us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:38.949675 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:38.950028 (+ 353us) client_negotiation.cc:769] Sending connection context
0810 03:42:38.950234 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 03:42:38.950704 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":146,"threads_started":1}
I20260810 03:42:38.954361 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:38.920226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:50853 (local address 127.21.175.126:39945)
0810 03:42:38.920948 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:38.920975 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:38.921459 (+ 484us) server_negotiation.cc:407] Connection header received
0810 03:42:38.924392 (+ 2933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:38.924413 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:38.924895 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:38.925312 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:38.927385 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:38.929159 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:38.931110 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:38.937668 (+ 6558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:38.942233 (+ 4565us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:38.942581 (+ 348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:38.942592 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:38.942619 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:38.942658 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:38.945264 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:38.948191 (+ 2927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:38.948200 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:38.948208 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:38.948376 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:38.948849 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:38.948853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:38.948855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:38.949414 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:38.952734 (+ 3320us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:38.952923 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 03:42:38.953296 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":150,"threads_started":1}
I20260810 03:42:38.957535 3675 heartbeater.cc:347] Connected to a master server at 127.21.175.126:39945
I20260810 03:42:38.957828 3675 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:38.958494 3675 heartbeater.cc:511] Master 127.21.175.126:39945 requested a full tablet report, sending...
W20260810 03:42:38.961018 3675 heartbeater.cc:644] Failed to heartbeat to 127.21.175.126:39945 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 03:42:38.961176 3675 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:38.961373 3675 heartbeater.cc:511] Master 127.21.175.126:39945 requested a full tablet report, sending...
I20260810 03:42:38.962497 3675 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:38.962687 3675 heartbeater.cc:511] Master 127.21.175.126:39945 requested a full tablet report, sending...
W20260810 03:42:38.963780 3675 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:42:39.098939 3679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:39.099323 3679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:39.099448 3679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:39.106545 3679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:39.106714 3679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:39.106855 3679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:42:39.114543 3679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.175.84:49917/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:40333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=42401
--tserver_master_addrs=127.21.175.126:39945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:39.116353 3679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:39.117985 3679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:39.132359 3688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:39.133649 3684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:39.133781 3685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:39.134076 3679 server_base.cc:1034] running on GCE node
I20260810 03:42:39.134629 3679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:39.135560 3679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:39.136727 3679 hybrid_clock.cc:648] HybridClock initialized: now 1786333359136684 us; error 57 us; skew 500 ppm
Aug 10 03:42:39 dist-test-slave-ss4n krb5kdc[2365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333359, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:39.140066 3679 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:42:39.141285 3679 webserver.cc:460] Webserver started at http://127.21.175.66:42401/ using document root <none> and password file <none>
I20260810 03:42:39.142066 3679 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:39.142191 3679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 03:42:39 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333359, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
I20260810 03:42:39.167029 3679 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.002s sys 0.001s
I20260810 03:42:39.170009 3697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:39.171680 3679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:42:39.171833 3679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "c032afb1882b4f428a9be96172c86403"
format_stamp: "Formatted at 2026-08-10 03:42:37 on dist-test-slave-ss4n"
server_key: "21db8997a34c019ae0fbb4c07b0a1fb1"
server_key_iv: "8d61cfc6572669fe65548362b6295acb"
server_key_version: "kuduclusterkey@0"
I20260810 03:42:39.172703 3679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:39.204077 3679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:39.207885 3679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:39.208184 3679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:39.209015 3679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:42:39.210162 3679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:42:39.210278 3679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:39.210379 3679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:42:39.210455 3679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:39.222040 3679 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:40333
I20260810 03:42:39.222126 3825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:40333 every 8 connection(s)
I20260810 03:42:39.223435 3679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:42:39.228416 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 3679
I20260810 03:42:39.228931 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.175.84:49917/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:35747
--local_ip_for_outbound_sockets=127.21.175.67
--tserver_master_addrs=127.21.175.126:39945
--webserver_port=35979
--webserver_interface=127.21.175.67
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.175.84:49917/kms with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:42:39 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333359, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:42:39.248055 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:39.225303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:54845 (local address 127.21.175.126:39945)
0810 03:42:39.225439 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:39.225444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:39.226429 (+ 985us) server_negotiation.cc:407] Connection header received
0810 03:42:39.227442 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:39.227445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:39.227521 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:39.227624 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:39.229028 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:39.229864 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:39.231646 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:39.232034 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:39.232153 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:39.243046 (+ 10893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:39.243052 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:39.243055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:39.243093 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:39.245014 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:39.245907 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:39.245911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:39.245912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:39.246085 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:39.246577 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:39.246582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:39.246585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:39.246846 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:39.246961 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:39.247714 (+ 753us) server_negotiation.cc:299] Negotiation successful
0810 03:42:39.247874 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:42:39.248952 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:39.225791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:39945 (local address 127.21.175.66:54845)
0810 03:42:39.226269 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:39.226323 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:39.227226 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:39.227809 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:39.227816 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:39.228097 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:39.228824 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:39.228848 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:39.230052 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:39.230056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:39.231492 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:39.231507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:39.232353 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:39.232356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:39.232559 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:39.233676 (+ 1117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:39.233701 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:39.242751 (+ 9050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:39.245189 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:39.245198 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:39.245223 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:39.245725 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:39.246203 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:39.246206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:39.246208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:39.246430 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:39.246969 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:39.246976 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:39.247348 (+ 372us) client_negotiation.cc:769] Sending connection context
0810 03:42:39.247562 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 03:42:39.247919 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":148,"threads_started":1}
I20260810 03:42:39.250630 3826 heartbeater.cc:347] Connected to a master server at 127.21.175.126:39945
I20260810 03:42:39.250880 3826 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:39.251608 3826 heartbeater.cc:511] Master 127.21.175.126:39945 requested a full tablet report, sending...
W20260810 03:42:39.253409 3826 heartbeater.cc:644] Failed to heartbeat to 127.21.175.126:39945 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 03:42:39.253582 3826 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:39.253777 3826 heartbeater.cc:511] Master 127.21.175.126:39945 requested a full tablet report, sending...
I20260810 03:42:39.254637 3826 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:39.254810 3826 heartbeater.cc:511] Master 127.21.175.126:39945 requested a full tablet report, sending...
W20260810 03:42:39.255717 3826 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:42:39.375136 3829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:39.375461 3829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:39.375568 3829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:39.380897 3829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:39.381042 3829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:39.381171 3829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:42:39.388062 3829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.175.84:49917/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:35747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=35979
--tserver_master_addrs=127.21.175.126:39945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:39.389554 3829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:39.390895 3829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:39.403467 3835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:39.405010 3829 server_base.cc:1034] running on GCE node
W20260810 03:42:39.410099 3839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:39.410799 3836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:39.411449 3829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:39.412341 3829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:39.413538 3829 hybrid_clock.cc:648] HybridClock initialized: now 1786333359413503 us; error 53 us; skew 500 ppm
Aug 10 03:42:39 dist-test-slave-ss4n krb5kdc[2365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333359, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:39.417358 3829 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:42:39.418676 3829 webserver.cc:460] Webserver started at http://127.21.175.67:35979/ using document root <none> and password file <none>
I20260810 03:42:39.419428 3829 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:39.419533 3829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 03:42:39 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333359, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
I20260810 03:42:39.444128 3829 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260810 03:42:39.450428 3848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:39.452286 3829 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 03:42:39.452436 3829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "a9740a0313924d839ded6b6e7dde6e89"
format_stamp: "Formatted at 2026-08-10 03:42:38 on dist-test-slave-ss4n"
server_key: "2ce104b628e9a95bdec7e466323b27c3"
server_key_iv: "77684ca5586f72319d4e4b08ed185719"
server_key_version: "kuduclusterkey@0"
I20260810 03:42:39.453266 3829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:39.471235 3829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:39.475301 3829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:39.475617 3829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:39.476370 3829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:42:39.477608 3829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:42:39.477723 3829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:39.477843 3829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:42:39.477921 3829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:39.490051 3829 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:35747
I20260810 03:42:39.491463 3829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:42:39.492794 3977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:35747 every 8 connection(s)
I20260810 03:42:39.499562 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 3829
Aug 10 03:42:39 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333359, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:42:39.507639 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:39.493544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:40973 (local address 127.21.175.126:39945)
0810 03:42:39.493683 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:39.493688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:39.494676 (+ 988us) server_negotiation.cc:407] Connection header received
0810 03:42:39.495868 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:39.495871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:39.495919 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:39.495992 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:39.497440 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:39.498152 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:39.499830 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:39.500113 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:39.500250 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:39.503194 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:39.503198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:39.503201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:39.503226 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:39.504774 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:39.505303 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:39.505308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:39.505309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:39.505396 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:39.505825 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:39.505828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:39.505830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:39.506023 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:39.506081 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:39.506778 (+ 697us) server_negotiation.cc:299] Negotiation successful
0810 03:42:39.506899 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 03:42:39.508234 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:39.493925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:39945 (local address 127.21.175.67:40973)
0810 03:42:39.494461 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:39.494535 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:39.495625 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:39.496184 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:39.496193 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:39.496558 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:39.497264 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:39.497286 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:39.498295 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:39.498299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:39.499689 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:39.499701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:39.500291 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:39.500296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:39.500543 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:39.501334 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:39.501351 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:39.503019 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:39.504899 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:39.504905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:39.504922 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:39.505196 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:39.505521 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:39.505524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:39.505526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:39.505720 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:39.506180 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:39.506189 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:39.506516 (+ 327us) client_negotiation.cc:769] Sending connection context
0810 03:42:39.506723 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 03:42:39.507053 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":118,"threads_started":1}
I20260810 03:42:39.512005 3978 heartbeater.cc:347] Connected to a master server at 127.21.175.126:39945
I20260810 03:42:39.512295 3978 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:39.512993 3978 heartbeater.cc:511] Master 127.21.175.126:39945 requested a full tablet report, sending...
I20260810 03:42:39.514801 3452 ts_manager.cc:194] Registered new tserver with Master: a9740a0313924d839ded6b6e7dde6e89 (127.21.175.67:35747)
I20260810 03:42:39.515964 3982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:39.500513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54472 (local address 127.21.175.126:39945)
0810 03:42:39.500911 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:39.500919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:39.500933 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:42:39.500968 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:39.500971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:39.501018 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:39.501114 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:39.502374 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:39.503613 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:39.505328 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:39.505711 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:39.505818 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:39.507836 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:39.507839 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:39.507842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:39.507865 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:39.509955 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:39.510612 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:39.510618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:39.510620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:39.510727 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:39.514864 (+ 4137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:39.514868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:39.514869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:39.515139 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:39.515325 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:39.515646 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 03:42:39.515789 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":183,"threads_started":1}
I20260810 03:42:39.516378 3452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:40973
Aug 10 03:42:39 dist-test-slave-ss4n krb5kdc[2365](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786333359, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:39.964008 3675 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:39.964325 3675 heartbeater.cc:511] Master 127.21.175.126:39945 requested a full tablet report, sending...
I20260810 03:42:39.965090 3452 ts_manager.cc:194] Registered new tserver with Master: ceda524dc1ca434cac517f9d49fc6678 (127.21.175.65:42217)
I20260810 03:42:39.965939 3452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:50853
Aug 10 03:42:40 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333359, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
I20260810 03:42:40.255973 3826 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:40.256289 3826 heartbeater.cc:511] Master 127.21.175.126:39945 requested a full tablet report, sending...
I20260810 03:42:40.257133 3452 ts_manager.cc:194] Registered new tserver with Master: c032afb1882b4f428a9be96172c86403 (127.21.175.66:40333)
I20260810 03:42:40.257915 3452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:54845
I20260810 03:42:40.269434 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:42:40.283066 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:40.271554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54480 (local address 127.21.175.126:39945)
0810 03:42:40.271833 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:40.271837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:40.272136 (+ 299us) server_negotiation.cc:407] Connection header received
0810 03:42:40.272177 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:40.272179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:40.272222 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:40.272327 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:40.273304 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.274128 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.275319 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.275656 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.275756 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:40.277734 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:40.277739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:40.277742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:40.277772 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:40.279818 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:40.280399 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:40.280403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:40.280404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:40.280552 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:40.281027 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:40.281031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:40.281032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:40.281284 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:40.281418 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:40.282491 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0810 03:42:40.282867 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
I20260810 03:42:40.288283 3452 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:42:40.293496 3452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:42:40.328517 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:40.320155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:40333 (local address 127.0.0.1:47256)
0810 03:42:40.322759 (+ 2604us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:40.322773 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:40.322877 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:40.323516 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:40.323520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:40.323538 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:42:40.323642 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:40.323647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.324966 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.324969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:40.326747 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:40.326756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.327958 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.327962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:40.328064 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:40.328106 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 03:42:40.328214 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 03:42:40.328348 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2528,"mutex_wait_us":612,"thread_start_us":101,"threads_started":1}
I20260810 03:42:40.328851 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:40.318952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:42217 (local address 127.0.0.1:53502)
0810 03:42:40.321785 (+ 2833us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:40.321795 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:40.321852 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:40.322486 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:40.322492 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:40.322516 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:42:40.322675 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:40.322684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.324893 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.324900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:40.326370 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:40.326380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.328433 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.328436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:40.328522 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:40.328558 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 03:42:40.328631 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 03:42:40.328702 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2782,"mutex_wait_us":57,"thread_start_us":101,"threads_started":1}
I20260810 03:42:40.329321 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:40.319376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.21.175.65:42217)
0810 03:42:40.322040 (+ 2664us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:40.322045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:40.322066 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:42:40.322131 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:40.322135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:40.322328 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:40.322498 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:42:40.322823 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.324764 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.326509 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.327323 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.327428 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:40.328374 (+ 946us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:40.329046 (+ 672us) server_negotiation.cc:299] Negotiation successful
0810 03:42:40.329182 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2577,"thread_start_us":120,"threads_started":1}
I20260810 03:42:40.329819 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:40.320610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47256 (local address 127.21.175.66:40333)
0810 03:42:40.321370 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:40.321380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:40.322971 (+ 1591us) server_negotiation.cc:407] Connection header received
0810 03:42:40.323053 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:40.323058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:40.323250 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:40.323381 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:42:40.323767 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.324851 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.326875 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.327649 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.327751 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:40.327898 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:40.329581 (+ 1683us) server_negotiation.cc:299] Negotiation successful
0810 03:42:40.329690 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":131,"threads_started":1}
I20260810 03:42:40.332652 3601 tablet_service.cc:1467] Processing CreateTablet for tablet 5a21dd7848894c54985fd571a9331542 (DEFAULT_TABLE table=test-table [id=bf6fdbc1a16c4c3099dc8e70fd7cc3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:42:40.332652 3752 tablet_service.cc:1467] Processing CreateTablet for tablet 5a21dd7848894c54985fd571a9331542 (DEFAULT_TABLE table=test-table [id=bf6fdbc1a16c4c3099dc8e70fd7cc3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:42:40.333866 3752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a21dd7848894c54985fd571a9331542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:42:40.334710 3601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a21dd7848894c54985fd571a9331542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:42:40.346203 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:40.317770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:35747 (local address 127.0.0.1:50642)
0810 03:42:40.321335 (+ 3565us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:40.321398 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:40.321589 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:40.328862 (+ 7273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:40.328868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:40.328893 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:42:40.329106 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:40.329114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.343014 (+ 13900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.343017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:40.344482 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:40.344495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.345700 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.345704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:40.345815 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:40.345842 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 03:42:40.345923 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 03:42:40.346017 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3362,"thread_start_us":122,"threads_started":1}
I20260810 03:42:40.347815 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:40.317891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.21.175.67:35747)
0810 03:42:40.325346 (+ 7455us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:40.325353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:40.325376 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:42:40.325449 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:40.325454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:40.325667 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:40.329276 (+ 3609us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:42:40.341587 (+ 12311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.342829 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.344682 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.345537 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.345678 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:40.345847 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:40.347515 (+ 1668us) server_negotiation.cc:299] Negotiation successful
0810 03:42:40.347651 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7343,"thread_start_us":97,"threads_started":1}
I20260810 03:42:40.348292 4009 tablet_bootstrap.cc:492] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403: Bootstrap starting.
I20260810 03:42:40.351007 4007 tablet_bootstrap.cc:492] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678: Bootstrap starting.
I20260810 03:42:40.351275 3907 tablet_service.cc:1467] Processing CreateTablet for tablet 5a21dd7848894c54985fd571a9331542 (DEFAULT_TABLE table=test-table [id=bf6fdbc1a16c4c3099dc8e70fd7cc3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:42:40.352473 3907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a21dd7848894c54985fd571a9331542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:42:40.353274 4007 tablet_bootstrap.cc:654] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678: Neither blocks nor log segments found. Creating new log.
I20260810 03:42:40.354182 4007 log.cc:824] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678: Log is configured to *not* fsync() on all Append() calls
I20260810 03:42:40.354919 4009 tablet_bootstrap.cc:654] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403: Neither blocks nor log segments found. Creating new log.
I20260810 03:42:40.355767 4009 log.cc:824] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403: Log is configured to *not* fsync() on all Append() calls
I20260810 03:42:40.358479 4007 tablet_bootstrap.cc:492] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678: No bootstrap required, opened a new log
I20260810 03:42:40.358721 4007 ts_tablet_manager.cc:1397] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 03:42:40.362937 4007 raft_consensus.cc:348] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9740a0313924d839ded6b6e7dde6e89" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 35747 } } peers { permanent_uuid: "ceda524dc1ca434cac517f9d49fc6678" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42217 } } peers { permanent_uuid: "c032afb1882b4f428a9be96172c86403" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 40333 } }
I20260810 03:42:40.363252 4007 raft_consensus.cc:374] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:42:40.363312 4007 raft_consensus.cc:729] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceda524dc1ca434cac517f9d49fc6678, State: Initialized, Role: FOLLOWER
I20260810 03:42:40.365955 4009 tablet_bootstrap.cc:492] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403: No bootstrap required, opened a new log
I20260810 03:42:40.366192 4009 ts_tablet_manager.cc:1397] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.001s
I20260810 03:42:40.369664 4009 raft_consensus.cc:348] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9740a0313924d839ded6b6e7dde6e89" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 35747 } } peers { permanent_uuid: "ceda524dc1ca434cac517f9d49fc6678" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42217 } } peers { permanent_uuid: "c032afb1882b4f428a9be96172c86403" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 40333 } }
I20260810 03:42:40.370007 4009 raft_consensus.cc:374] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:42:40.370065 4009 raft_consensus.cc:729] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c032afb1882b4f428a9be96172c86403, State: Initialized, Role: FOLLOWER
I20260810 03:42:40.370165 4007 consensus_queue.cc:260] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9740a0313924d839ded6b6e7dde6e89" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 35747 } } peers { permanent_uuid: "ceda524dc1ca434cac517f9d49fc6678" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42217 } } peers { permanent_uuid: "c032afb1882b4f428a9be96172c86403" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 40333 } }
I20260810 03:42:40.370604 4009 consensus_queue.cc:260] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9740a0313924d839ded6b6e7dde6e89" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 35747 } } peers { permanent_uuid: "ceda524dc1ca434cac517f9d49fc6678" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42217 } } peers { permanent_uuid: "c032afb1882b4f428a9be96172c86403" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 40333 } }
I20260810 03:42:40.371706 4009 ts_tablet_manager.cc:1428] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260810 03:42:40.371706 4007 ts_tablet_manager.cc:1428] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260810 03:42:40.372107 3826 heartbeater.cc:503] Master 127.21.175.126:39945 was elected leader, sending a full tablet report...
I20260810 03:42:40.372130 3675 heartbeater.cc:503] Master 127.21.175.126:39945 was elected leader, sending a full tablet report...
I20260810 03:42:40.372520 4012 tablet_bootstrap.cc:492] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89: Bootstrap starting.
I20260810 03:42:40.375172 4012 tablet_bootstrap.cc:654] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89: Neither blocks nor log segments found. Creating new log.
I20260810 03:42:40.376163 4012 log.cc:824] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89: Log is configured to *not* fsync() on all Append() calls
I20260810 03:42:40.380200 4012 tablet_bootstrap.cc:492] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89: No bootstrap required, opened a new log
I20260810 03:42:40.380427 4012 ts_tablet_manager.cc:1397] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260810 03:42:40.386461 4012 raft_consensus.cc:348] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9740a0313924d839ded6b6e7dde6e89" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 35747 } } peers { permanent_uuid: "ceda524dc1ca434cac517f9d49fc6678" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42217 } } peers { permanent_uuid: "c032afb1882b4f428a9be96172c86403" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 40333 } }
I20260810 03:42:40.386852 4012 raft_consensus.cc:374] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:42:40.386921 4012 raft_consensus.cc:729] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9740a0313924d839ded6b6e7dde6e89, State: Initialized, Role: FOLLOWER
I20260810 03:42:40.387494 4012 consensus_queue.cc:260] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9740a0313924d839ded6b6e7dde6e89" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 35747 } } peers { permanent_uuid: "ceda524dc1ca434cac517f9d49fc6678" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42217 } } peers { permanent_uuid: "c032afb1882b4f428a9be96172c86403" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 40333 } }
I20260810 03:42:40.388301 3978 heartbeater.cc:503] Master 127.21.175.126:39945 was elected leader, sending a full tablet report...
I20260810 03:42:40.389420 4012 ts_tablet_manager.cc:1428] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89: Time spent starting tablet: real 0.009s user 0.009s sys 0.001s
W20260810 03:42:40.420149 3676 tablet.cc:2369] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:42:40.475238 3827 tablet.cc:2369] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:42:40.493230 3979 tablet.cc:2369] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 03:42:40 dist-test-slave-ss4n krb5kdc[2365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333359, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for HTTP/127.21.175.84@KRBTEST.COM
I20260810 03:42:40.556555 4013 raft_consensus.cc:482] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:42:40.556849 4013 raft_consensus.cc:504] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9740a0313924d839ded6b6e7dde6e89" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 35747 } } peers { permanent_uuid: "ceda524dc1ca434cac517f9d49fc6678" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42217 } } peers { permanent_uuid: "c032afb1882b4f428a9be96172c86403" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 40333 } }
I20260810 03:42:40.558184 4013 leader_election.cc:288] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9740a0313924d839ded6b6e7dde6e89 (127.21.175.67:35747), c032afb1882b4f428a9be96172c86403 (127.21.175.66:40333)
I20260810 03:42:40.569253 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:40.559470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:37515 (local address 127.21.175.67:35747)
0810 03:42:40.559652 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:40.559656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:40.559672 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:42:40.559729 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:40.559732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:40.559778 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:40.559859 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:42:40.565502 (+ 5643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.566252 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.567709 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.568453 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.568553 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:40.568602 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:40.569014 (+ 412us) server_negotiation.cc:299] Negotiation successful
0810 03:42:40.569108 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 03:42:40.569609 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:40.559646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:55733 (local address 127.21.175.66:40333)
0810 03:42:40.560046 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:40.560050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:40.560590 (+ 540us) server_negotiation.cc:407] Connection header received
0810 03:42:40.560816 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:40.560819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:40.560867 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:40.560913 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:42:40.562126 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.562937 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.567163 (+ 4226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.567740 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.567850 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:40.567912 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:40.569363 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0810 03:42:40.569458 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312}
I20260810 03:42:40.569746 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:40.558374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:35747 (local address 127.21.175.65:37515)
0810 03:42:40.559157 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:40.559174 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:40.559289 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:40.559889 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:40.559892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:40.559923 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:42:40.560270 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:40.560281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.566416 (+ 6135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.566419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:40.567548 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:40.567560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.568698 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.568701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:40.568802 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:40.568829 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 03:42:40.569464 (+ 635us) client_negotiation.cc:240] Negotiation successful
0810 03:42:40.569571 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":142,"threads_started":1}
I20260810 03:42:40.570959 3911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a21dd7848894c54985fd571a9331542" candidate_uuid: "ceda524dc1ca434cac517f9d49fc6678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9740a0313924d839ded6b6e7dde6e89" is_pre_election: true
I20260810 03:42:40.571319 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:40.558832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:40333 (local address 127.21.175.65:55733)
0810 03:42:40.560486 (+ 1654us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:40.560501 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:40.560688 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:40.561556 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:40.561559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:40.561580 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:42:40.561970 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:40.561982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.563834 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.563837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:40.566608 (+ 2771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:40.566622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.568751 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.568763 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:40.568887 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:40.569229 (+ 342us) client_negotiation.cc:769] Sending connection context
0810 03:42:40.570968 (+ 1739us) client_negotiation.cc:240] Negotiation successful
0810 03:42:40.571057 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1296,"thread_start_us":144,"threads_started":1}
I20260810 03:42:40.571346 3911 raft_consensus.cc:2454] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceda524dc1ca434cac517f9d49fc6678 in term 0.
I20260810 03:42:40.571879 3551 leader_election.cc:302] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9740a0313924d839ded6b6e7dde6e89, ceda524dc1ca434cac517f9d49fc6678; no voters:
I20260810 03:42:40.572417 4013 raft_consensus.cc:2781] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:42:40.572508 4013 raft_consensus.cc:482] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:42:40.572551 4013 raft_consensus.cc:3037] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:42:40.572646 3757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a21dd7848894c54985fd571a9331542" candidate_uuid: "ceda524dc1ca434cac517f9d49fc6678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c032afb1882b4f428a9be96172c86403" is_pre_election: true
I20260810 03:42:40.572918 3757 raft_consensus.cc:2454] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceda524dc1ca434cac517f9d49fc6678 in term 0.
I20260810 03:42:40.574180 4013 raft_consensus.cc:504] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9740a0313924d839ded6b6e7dde6e89" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 35747 } } peers { permanent_uuid: "ceda524dc1ca434cac517f9d49fc6678" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42217 } } peers { permanent_uuid: "c032afb1882b4f428a9be96172c86403" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 40333 } }
I20260810 03:42:40.574585 4013 leader_election.cc:288] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [CANDIDATE]: Term 1 election: Requested vote from peers a9740a0313924d839ded6b6e7dde6e89 (127.21.175.67:35747), c032afb1882b4f428a9be96172c86403 (127.21.175.66:40333)
I20260810 03:42:40.575160 3911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a21dd7848894c54985fd571a9331542" candidate_uuid: "ceda524dc1ca434cac517f9d49fc6678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9740a0313924d839ded6b6e7dde6e89"
I20260810 03:42:40.575304 3911 raft_consensus.cc:3037] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:42:40.576493 3911 raft_consensus.cc:2454] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceda524dc1ca434cac517f9d49fc6678 in term 1.
I20260810 03:42:40.576800 3757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a21dd7848894c54985fd571a9331542" candidate_uuid: "ceda524dc1ca434cac517f9d49fc6678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c032afb1882b4f428a9be96172c86403"
I20260810 03:42:40.576859 3551 leader_election.cc:302] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9740a0313924d839ded6b6e7dde6e89, ceda524dc1ca434cac517f9d49fc6678; no voters:
I20260810 03:42:40.576939 3757 raft_consensus.cc:3037] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:42:40.577116 4013 raft_consensus.cc:2781] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:42:40.577379 4013 raft_consensus.cc:686] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [term 1 LEADER]: Becoming Leader. State: Replica: ceda524dc1ca434cac517f9d49fc6678, State: Running, Role: LEADER
I20260810 03:42:40.577776 4013 consensus_queue.cc:237] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9740a0313924d839ded6b6e7dde6e89" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 35747 } } peers { permanent_uuid: "ceda524dc1ca434cac517f9d49fc6678" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42217 } } peers { permanent_uuid: "c032afb1882b4f428a9be96172c86403" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 40333 } }
I20260810 03:42:40.578266 3757 raft_consensus.cc:2454] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceda524dc1ca434cac517f9d49fc6678 in term 1.
I20260810 03:42:40.581780 3452 catalog_manager.cc:5495] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 reported cstate change: term changed from 0 to 1, leader changed from <none> to ceda524dc1ca434cac517f9d49fc6678 (127.21.175.65). New cstate: current_term: 1 leader_uuid: "ceda524dc1ca434cac517f9d49fc6678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9740a0313924d839ded6b6e7dde6e89" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 35747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceda524dc1ca434cac517f9d49fc6678" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c032afb1882b4f428a9be96172c86403" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 40333 } health_report { overall_health: UNKNOWN } } }
I20260810 03:42:40.625068 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:40.619138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53504 (local address 127.21.175.65:42217)
0810 03:42:40.619303 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:40.619308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:40.619329 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:42:40.619434 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:40.619438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:40.619516 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:40.619653 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:42:40.620125 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.621028 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.622581 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:40.623003 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:40.623109 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:40.623625 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:42:40.624232 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:42:40.624377 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:40.624797 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 03:42:40.624868 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:42:40.646626 3757 raft_consensus.cc:1261] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403 [term 1 FOLLOWER]: Refusing update from remote peer ceda524dc1ca434cac517f9d49fc6678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:42:40.647504 4020 consensus_queue.cc:1035] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c032afb1882b4f428a9be96172c86403" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 40333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:42:40.655848 3911 raft_consensus.cc:1261] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89 [term 1 FOLLOWER]: Refusing update from remote peer ceda524dc1ca434cac517f9d49fc6678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:42:40.656620 4020 consensus_queue.cc:1035] T 5a21dd7848894c54985fd571a9331542 P ceda524dc1ca434cac517f9d49fc6678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9740a0313924d839ded6b6e7dde6e89" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 35747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:42:40.682737 4024 mvcc.cc:205] Tried to move back new op lower bound from 7316821445176856576 to 7316821444930961408. Current Snapshot: MvccSnapshot[applied={T|T < 7316821445176856576}]
I20260810 03:42:40.686074 4025 mvcc.cc:205] Tried to move back new op lower bound from 7316821445176856576 to 7316821444930961408. Current Snapshot: MvccSnapshot[applied={T|T < 7316821445176856576}]
I20260810 03:42:40.696516 3452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:42:40.696873 3452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:42:40.700309 3452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26eff2c679fb4fc1a483881229f47be8: Sending DeleteTablet for 3 replicas of tablet 5a21dd7848894c54985fd571a9331542
I20260810 03:42:40.701834 3907 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a21dd7848894c54985fd571a9331542 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:42:40 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:50642
I20260810 03:42:40.702046 3752 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a21dd7848894c54985fd571a9331542 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:42:40 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:47256
I20260810 03:42:40.702541 3601 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a21dd7848894c54985fd571a9331542 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:42:40 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:53502
I20260810 03:42:40.703159 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 3493
I20260810 03:42:40.711284 4036 tablet_replica.cc:330] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403: stopping tablet replica
I20260810 03:42:40.717829 4036 raft_consensus.cc:2229] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:42:40.718336 4036 raft_consensus.cc:2258] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:42:40.722275 4036 ts_tablet_manager.cc:1905] T 5a21dd7848894c54985fd571a9331542 P c032afb1882b4f428a9be96172c86403: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 03:42:40.725266 3438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.175.65:42217 (error 108)
I20260810 03:42:40.726792 4035 tablet_replica.cc:330] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89: stopping tablet replica
I20260810 03:42:40.727198 4035 raft_consensus.cc:2229] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:42:40.727618 4035 raft_consensus.cc:2258] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:42:40.730000 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 3679
I20260810 03:42:40.735491 4035 ts_tablet_manager.cc:1905] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:42:40.738137 4035 ts_tablet_manager.cc:1918] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:42:40.738307 4035 log.cc:1196] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/5a21dd7848894c54985fd571a9331542
I20260810 03:42:40.738726 4035 ts_tablet_manager.cc:1939] T 5a21dd7848894c54985fd571a9331542 P a9740a0313924d839ded6b6e7dde6e89: Deleting consensus metadata
I20260810 03:42:40.740340 3437 catalog_manager.cc:4841] TS a9740a0313924d839ded6b6e7dde6e89 (127.21.175.67:35747): tablet 5a21dd7848894c54985fd571a9331542 (table test-table [id=bf6fdbc1a16c4c3099dc8e70fd7cc3d4]) successfully deleted
I20260810 03:42:40.746547 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:40.746156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:42217 (local address 127.0.0.1:53518)
0810 03:42:40.746343 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:40.746443 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:42217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260810 03:42:40.746927 3438 catalog_manager.cc:4584] TS ceda524dc1ca434cac517f9d49fc6678 (127.21.175.65:42217): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a21dd7848894c54985fd571a9331542: Network error: Client connection negotiation failed: client connection to 127.21.175.65:42217: connect: Connection refused (error 111)
I20260810 03:42:40.752075 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 3829
I20260810 03:42:40.752583 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:40.752181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:40333 (local address 127.0.0.1:47270)
0810 03:42:40.752428 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:40.752486 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.66:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260810 03:42:40.763960 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 3418
I20260810 03:42:40.773768 22205 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 03:42:41.119676 22205 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 03:42:41.119798 22205 mini_ranger.cc:66] Stopping Ranger...
I20260810 03:42:41.465739 22205 mini_ranger.cc:68] Stopped Ranger
2026-08-10 03:42:41.465 UTC [2396] LOG: received smart shutdown request
2026-08-10 03:42:41.470 UTC [2396] LOG: background worker "logical replication launcher" (PID 2405) exited with exit code 1
2026-08-10 03:42:41.470 UTC [2400] LOG: shutting down
2026-08-10 03:42:41.666 UTC [2396] LOG: database system is shut down
2026-08-10T03:42:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51975 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:42:41 dist-test-slave-ss4n krb5kdc[4049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:42:41 dist-test-slave-ss4n krb5kdc[4049](info): set up 2 sockets
Aug 10 03:42:41 dist-test-slave-ss4n krb5kdc[4049](info): commencing operation
krb5kdc: starting...
W20260810 03:42:42.981743 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:42:43 dist-test-slave-ss4n krb5kdc[4049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:42:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:42:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:42:43.127278 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:44713
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:44713,127.21.175.125:44799,127.21.175.124:40187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:42:43.244239 4065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:43.244576 4065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:43.244652 4065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:43.248790 4065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:42:43.248903 4065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:43.248939 4065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:43.248980 4065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:42:43.249015 4065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:42:43.254707 4065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:44713,127.21.175.125:44799,127.21.175.124:40187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:44713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:43.255955 4065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:43.257041 4065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:43.266253 4070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:43.266182 4073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:43.266331 4065 server_base.cc:1034] running on GCE node
W20260810 03:42:43.266182 4071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:43.267176 4065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:43.268293 4065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:43.269588 4065 hybrid_clock.cc:648] HybridClock initialized: now 1786333363269559 us; error 36 us; skew 500 ppm
Aug 10 03:42:43 dist-test-slave-ss4n krb5kdc[4049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333363, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:43.272636 4065 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:42:43.273654 4065 webserver.cc:460] Webserver started at http://127.21.175.126:44715/ using document root <none> and password file <none>
I20260810 03:42:43.274291 4065 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:43.274358 4065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:43.274538 4065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:42:43.276808 4065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "541a6c8d7fa44b2684cacd4c615566e4"
format_stamp: "Formatted at 2026-08-10 03:42:43 on dist-test-slave-ss4n"
server_key: "ea43d93c23337e6e3d80cd378c188329"
server_key_iv: "93ab1e50b1ce4c50cf9011533b9042fe"
server_key_version: "encryptionkey@0"
I20260810 03:42:43.277354 4065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "541a6c8d7fa44b2684cacd4c615566e4"
format_stamp: "Formatted at 2026-08-10 03:42:43 on dist-test-slave-ss4n"
server_key: "ea43d93c23337e6e3d80cd378c188329"
server_key_iv: "93ab1e50b1ce4c50cf9011533b9042fe"
server_key_version: "encryptionkey@0"
I20260810 03:42:43.281299 4065 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 03:42:43.284080 4081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:43.285658 4065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:42:43.285806 4065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "541a6c8d7fa44b2684cacd4c615566e4"
format_stamp: "Formatted at 2026-08-10 03:42:43 on dist-test-slave-ss4n"
server_key: "ea43d93c23337e6e3d80cd378c188329"
server_key_iv: "93ab1e50b1ce4c50cf9011533b9042fe"
server_key_version: "encryptionkey@0"
I20260810 03:42:43.285930 4065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:43.309262 4065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:43.314734 4065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:43.314976 4065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:43.325649 4065 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:44713
I20260810 03:42:43.326035 4143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:44713 every 8 connection(s)
I20260810 03:42:43.326794 4065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:42:43.330185 4145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:42:43.335610 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 4065
I20260810 03:42:43.335768 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:42:43.336067 22205 external_mini_cluster.cc:1428] Setting key c069f3160919544417aae71da632a903
I20260810 03:42:43.335902 4145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.175.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.125@KRBTEST.COM" created.
Aug 10 03:42:43 dist-test-slave-ss4n krb5kdc[4049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333363, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:42:43.358752 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.338697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:44713 (local address 127.0.0.1:60048)
0810 03:42:43.339235 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.339291 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:43.340250 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:43.341018 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:43.341024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:43.341473 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:43.341862 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.341883 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.344004 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.344007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.345296 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.345308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.346482 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.346486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.346620 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.347304 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:43.347328 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:43.352374 (+ 5046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:43.355155 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.355167 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.355181 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.355450 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.355834 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.355837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.355839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.355978 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.356707 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:43.356714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:43.356871 (+ 157us) client_negotiation.cc:769] Sending connection context
0810 03:42:43.357107 (+ 236us) client_negotiation.cc:240] Negotiation successful
0810 03:42:43.357577 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":121,"thread_start_us":163,"threads_started":1}
I20260810 03:42:43.358752 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.338698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.21.175.126:44713)
0810 03:42:43.339416 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:43.339429 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:43.339482 (+ 53us) server_negotiation.cc:407] Connection header received
0810 03:42:43.340593 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:43.340598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:43.340785 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:43.341016 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:43.342058 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.343875 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.345412 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.346053 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.346223 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.352629 (+ 6406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:43.352640 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:43.352660 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:43.352697 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:43.354997 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.355582 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.355588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.355590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.355718 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.356086 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.356091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.356093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.356567 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:43.356730 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:43.357191 (+ 461us) server_negotiation.cc:299] Negotiation successful
0810 03:42:43.357484 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":587,"spinlock_wait_cycles":8960,"thread_start_us":163,"threads_started":1}
I20260810 03:42:43.362087 4145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:42:43.362993 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.362679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:44799 (local address 127.0.0.1:49208)
0810 03:42:43.362815 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.362909 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:44799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 03:42:43.363278 4082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.175.125:44799: connect: Connection refused (error 111)
I20260810 03:42:43.364336 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.364067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:44799 (local address 127.0.0.1:49224)
0810 03:42:43.364189 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.364251 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:44799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 03:42:43.364499 4145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.125:44799: Network error: Client connection negotiation failed: client connection to 127.21.175.125:44799: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.125.keytab.
Entry for principal kudu/127.21.175.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.125.keytab.
I20260810 03:42:43.373853 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.125:44799
--webserver_interface=127.21.175.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.125.keytab
--principal=kudu/127.21.175.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:44713,127.21.175.125:44799,127.21.175.124:40187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:42:43.413794 4145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } attempt: 1
I20260810 03:42:43.414829 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.414475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:44799 (local address 127.0.0.1:49236)
0810 03:42:43.414674 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.414744 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:44799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 03:42:43.415782 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.415450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:44799 (local address 127.0.0.1:49244)
0810 03:42:43.415662 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.415722 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:44799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260810 03:42:43.415915 4145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.125:44799: Network error: Client connection negotiation failed: client connection to 127.21.175.125:44799: connect: Connection refused (error 111)
I20260810 03:42:43.484131 4145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } attempt: 2
I20260810 03:42:43.485162 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.484771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:44799 (local address 127.0.0.1:49256)
0810 03:42:43.484995 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.485073 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:44799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 03:42:43.486174 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.485881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:44799 (local address 127.0.0.1:49258)
0810 03:42:43.486020 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.486085 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:44799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 03:42:43.486321 4145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.125:44799: Network error: Client connection negotiation failed: client connection to 127.21.175.125:44799: connect: Connection refused (error 111)
W20260810 03:42:43.493407 4155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:43.493768 4155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:43.493844 4155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:43.497967 4155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:42:43.498029 4155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:43.498050 4155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:43.498068 4155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:42:43.498083 4155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:42:43.503773 4155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:44713,127.21.175.125:44799,127.21.175.124:40187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.125.keytab
--principal=kudu/127.21.175.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.125:44799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.175.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:43.504931 4155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:43.506055 4155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:43.515012 4162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:43.515031 4160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:43.515012 4164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:43.515626 4155 server_base.cc:1034] running on GCE node
I20260810 03:42:43.516219 4155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:43.517532 4155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:43.518743 4155 hybrid_clock.cc:648] HybridClock initialized: now 1786333363518723 us; error 28 us; skew 500 ppm
Aug 10 03:42:43 dist-test-slave-ss4n krb5kdc[4049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333363, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:43.522011 4155 init.cc:377] Logged in from keytab as kudu/127.21.175.125@KRBTEST.COM (short username kudu)
I20260810 03:42:43.523407 4155 webserver.cc:460] Webserver started at http://127.21.175.125:39869/ using document root <none> and password file <none>
I20260810 03:42:43.524104 4155 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:43.524209 4155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:43.524454 4155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:42:43.526841 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-1/data/instance:
uuid: "bf4c34f3e57540ddad79abb54729423d"
format_stamp: "Formatted at 2026-08-10 03:42:43 on dist-test-slave-ss4n"
server_key: "c476046af1ad3654cd7d75963465fe59"
server_key_iv: "5002fe241c932f01687394a0bd2f1de6"
server_key_version: "encryptionkey@0"
I20260810 03:42:43.527493 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-1/wal/instance:
uuid: "bf4c34f3e57540ddad79abb54729423d"
format_stamp: "Formatted at 2026-08-10 03:42:43 on dist-test-slave-ss4n"
server_key: "c476046af1ad3654cd7d75963465fe59"
server_key_iv: "5002fe241c932f01687394a0bd2f1de6"
server_key_version: "encryptionkey@0"
I20260810 03:42:43.532003 4155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:42:43.534981 4171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:43.536406 4155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:42:43.536675 4155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-1/wal
uuid: "bf4c34f3e57540ddad79abb54729423d"
format_stamp: "Formatted at 2026-08-10 03:42:43 on dist-test-slave-ss4n"
server_key: "c476046af1ad3654cd7d75963465fe59"
server_key_iv: "5002fe241c932f01687394a0bd2f1de6"
server_key_version: "encryptionkey@0"
I20260810 03:42:43.536826 4155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:43.558277 4155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:43.561909 4155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:43.562175 4155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:43.570710 4155 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.125:44799
I20260810 03:42:43.571003 4234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.125:44799 every 8 connection(s)
I20260810 03:42:43.571851 4155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-1/data/info.pb
I20260810 03:42:43.574878 4235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:42:43.577558 4145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } attempt: 3
I20260810 03:42:43.578989 4235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:42:43.582161 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 4155
I20260810 03:42:43.582296 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-1/wal/instance
I20260810 03:42:43.582626 22205 external_mini_cluster.cc:1428] Setting key ee5c2e40db871c7ee7575fbc1e4fd473
Aug 10 03:42:43 dist-test-slave-ss4n krb5kdc[4049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333363, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for kudu/127.21.175.125@KRBTEST.COM
Aug 10 03:42:43 dist-test-slave-ss4n krb5kdc[4049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333363, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.125@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.175.124@KRBTEST.COM; defaulting to no policy
I20260810 03:42:43.593320 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.578097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:44799 (local address 127.0.0.1:49268)
0810 03:42:43.578243 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.578261 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:43.578363 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:43.580973 (+ 2610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:43.580981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:43.581232 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:43.581443 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.581455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.583380 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.583385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.584326 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.584338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.585465 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.585468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.585640 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.586295 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:43.586317 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:43.588047 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:43.590754 (+ 2707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.590759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.590763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.591024 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.591471 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.591486 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.591488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.591595 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.592662 (+ 1067us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:43.592666 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:43.592778 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 03:42:43.592897 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 03:42:43.593075 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"spinlock_wait_cycles":10112}
I20260810 03:42:43.594698 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.578846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49268 (local address 127.21.175.125:44799)
0810 03:42:43.579320 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:43.579342 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:43.579389 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:42:43.580205 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:43.580227 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:43.580630 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:43.580904 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:43.581684 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.583160 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.584472 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.585289 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.585470 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.588227 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:43.588238 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:43.588260 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:43.588296 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:43.590626 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.591162 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.591173 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.591182 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.591346 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.591749 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.591753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.591755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.592520 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:43.592686 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:43.593502 (+ 816us) server_negotiation.cc:299] Negotiation successful
0810 03:42:43.593804 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":319,"thread_start_us":127,"threads_started":1}
I20260810 03:42:43.594874 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.581138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.21.175.126:44713)
0810 03:42:43.581255 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:43.581259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:43.581823 (+ 564us) server_negotiation.cc:407] Connection header received
0810 03:42:43.582020 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:43.582025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:43.582075 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:43.582130 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:43.583323 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.584253 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.585712 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.586124 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.586227 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.589686 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:43.589692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:43.589699 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:43.589733 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:43.592744 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.593320 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.593324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.593326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.593430 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.593813 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.593816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.593818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.594020 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:43.594270 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:43.594559 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 03:42:43.594690 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":51,"spinlock_wait_cycles":1920}
I20260810 03:42:43.594901 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.581113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:44713 (local address 127.0.0.1:60062)
0810 03:42:43.581695 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.581737 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:43.581894 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:43.582362 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:43.582368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:43.582893 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:43.583142 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.583155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.584451 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.584457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.585537 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.585548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.586400 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.586405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.586515 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.587153 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:43.587173 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:43.589516 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:43.592866 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.592877 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.592884 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.593199 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.593589 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.593593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.593595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.593693 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.594151 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:43.594158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:43.594298 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 03:42:43.594482 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 03:42:43.594691 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":371,"threads_started":1}
I20260810 03:42:43.596199 4235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.124@KRBTEST.COM" created.
I20260810 03:42:43.596971 4145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:42:43.597805 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.597464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:40187 (local address 127.0.0.1:39550)
0810 03:42:43.597653 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.597718 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260810 03:42:43.598556 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.598272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:40187 (local address 127.0.0.1:39552)
0810 03:42:43.598424 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.598476 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260810 03:42:43.598695 4145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:40187: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
Aug 10 03:42:43 dist-test-slave-ss4n krb5kdc[4049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333363, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.125@KRBTEST.COM for kudu/127.21.175.125@KRBTEST.COM
I20260810 03:42:43.606506 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.596823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:44799 (local address 127.0.0.1:49274)
0810 03:42:43.597084 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.597099 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:43.597186 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:43.598048 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:43.598054 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:43.598255 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:43.598404 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.598415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.599325 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.599330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.600205 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.600219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.600688 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.600695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.600830 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.601351 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:43.601370 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:43.602885 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:43.604935 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.604937 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.604939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.605152 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.605456 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.605458 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.605459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.605625 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.606096 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:43.606098 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:43.606178 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 03:42:43.606262 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 03:42:43.606374 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260810 03:42:43.606649 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.597092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.21.175.125:44799)
0810 03:42:43.597653 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:43.597659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:43.597676 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:42:43.597714 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:43.597717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:43.597761 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:43.597854 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:43.598540 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.599211 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.600339 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.600572 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.600654 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.603041 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:43.603054 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:43.603058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:43.603092 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:43.604811 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.605259 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.605265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.605268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.605357 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.605724 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.605728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.605730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.605988 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:43.606106 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:43.606359 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 03:42:43.606502 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482}
I20260810 03:42:43.607391 4235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:42:43.608139 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.607823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:40187 (local address 127.0.0.1:39568)
0810 03:42:43.607972 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.608078 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260810 03:42:43.608999 4176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
I20260810 03:42:43.610121 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.609882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:40187 (local address 127.0.0.1:39580)
0810 03:42:43.610016 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.610066 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 03:42:43.610492 4235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:40187: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.124.keytab.
Entry for principal kudu/127.21.175.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.124.keytab.
I20260810 03:42:43.617086 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.124:40187
--webserver_interface=127.21.175.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.124.keytab
--principal=kudu/127.21.175.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:44713,127.21.175.125:44799,127.21.175.124:40187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:42:43.629859 4145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } attempt: 1
I20260810 03:42:43.630812 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.630455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:40187 (local address 127.0.0.1:39590)
0810 03:42:43.630659 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.630732 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260810 03:42:43.631738 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.631419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:40187 (local address 127.0.0.1:39592)
0810 03:42:43.631597 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.631674 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 03:42:43.631954 4145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:40187: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
I20260810 03:42:43.659817 4235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } attempt: 1
I20260810 03:42:43.660787 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.660466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:40187 (local address 127.0.0.1:39596)
0810 03:42:43.660643 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.660708 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 03:42:43.661607 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.661329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:40187 (local address 127.0.0.1:39612)
0810 03:42:43.661441 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.661542 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 03:42:43.661734 4235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:40187: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
I20260810 03:42:43.707180 4145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } attempt: 2
I20260810 03:42:43.708228 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.707845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:40187 (local address 127.0.0.1:39620)
0810 03:42:43.708053 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.708123 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 03:42:43.709079 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.708831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:40187 (local address 127.0.0.1:39628)
0810 03:42:43.708953 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.709019 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 03:42:43.709214 4145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:40187: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
I20260810 03:42:43.729919 4235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } attempt: 2
I20260810 03:42:43.730872 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.730538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:40187 (local address 127.0.0.1:39634)
0810 03:42:43.730720 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.730791 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 03:42:43.731809 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.731515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:40187 (local address 127.0.0.1:39644)
0810 03:42:43.731682 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.731737 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 03:42:43.731968 4235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:40187: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
W20260810 03:42:43.742463 4242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:43.742789 4242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:43.742890 4242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:43.747395 4242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:42:43.747495 4242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:43.747534 4242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:43.747594 4242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:42:43.747650 4242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:42:43.753275 4242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:44713,127.21.175.125:44799,127.21.175.124:40187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.124.keytab
--principal=kudu/127.21.175.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.124:40187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.175.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:43.754547 4242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:43.755635 4242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:43.764840 4248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:43.764859 4247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:43.764860 4251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:43.765286 4242 server_base.cc:1034] running on GCE node
I20260810 03:42:43.765934 4242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:43.767185 4242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:43.768400 4242 hybrid_clock.cc:648] HybridClock initialized: now 1786333363768362 us; error 55 us; skew 500 ppm
Aug 10 03:42:43 dist-test-slave-ss4n krb5kdc[4049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333363, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:43.771471 4242 init.cc:377] Logged in from keytab as kudu/127.21.175.124@KRBTEST.COM (short username kudu)
I20260810 03:42:43.772487 4242 webserver.cc:460] Webserver started at http://127.21.175.124:38927/ using document root <none> and password file <none>
I20260810 03:42:43.773075 4242 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:43.773176 4242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:43.773423 4242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:42:43.775766 4242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-2/data/instance:
uuid: "6ac6546f7ea6475eb84ecf952395c517"
format_stamp: "Formatted at 2026-08-10 03:42:43 on dist-test-slave-ss4n"
server_key: "8b15c3cdaf38d677b699b025c625de95"
server_key_iv: "77019bf95f97f6f1b90a3187a1202b14"
server_key_version: "encryptionkey@0"
I20260810 03:42:43.776364 4242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-2/wal/instance:
uuid: "6ac6546f7ea6475eb84ecf952395c517"
format_stamp: "Formatted at 2026-08-10 03:42:43 on dist-test-slave-ss4n"
server_key: "8b15c3cdaf38d677b699b025c625de95"
server_key_iv: "77019bf95f97f6f1b90a3187a1202b14"
server_key_version: "encryptionkey@0"
I20260810 03:42:43.780443 4242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 03:42:43.783277 4260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:43.784737 4242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260810 03:42:43.784888 4242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-2/wal
uuid: "6ac6546f7ea6475eb84ecf952395c517"
format_stamp: "Formatted at 2026-08-10 03:42:43 on dist-test-slave-ss4n"
server_key: "8b15c3cdaf38d677b699b025c625de95"
server_key_iv: "77019bf95f97f6f1b90a3187a1202b14"
server_key_version: "encryptionkey@0"
I20260810 03:42:43.785008 4242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:43.801745 4242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:43.805015 4242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:43.805231 4242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:43.808447 4145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } attempt: 3
I20260810 03:42:43.809484 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.809110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:40187 (local address 127.0.0.1:39656)
0810 03:42:43.809283 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.809354 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 03:42:43.810453 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.810174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:40187 (local address 127.0.0.1:39672)
0810 03:42:43.810305 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.810367 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 03:42:43.810652 4145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:40187: Network error: Client connection negotiation failed: client connection to 127.21.175.124:40187: connect: Connection refused (error 111)
I20260810 03:42:43.813027 4242 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.124:40187
I20260810 03:42:43.813560 4316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.124:40187 every 8 connection(s)
I20260810 03:42:43.814182 4242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-2/data/info.pb
I20260810 03:42:43.814960 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 4242
I20260810 03:42:43.815099 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333190020696-22205-0/minicluster-data/master-2/wal/instance
I20260810 03:42:43.815387 22205 external_mini_cluster.cc:1428] Setting key a13fe9e78512fc5d9cb39a0fec0ff4bf
I20260810 03:42:43.817679 4317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:42:43.822422 4317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:42:43.823143 4235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } attempt: 3
Aug 10 03:42:43 dist-test-slave-ss4n krb5kdc[4049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.124@KRBTEST.COM
Aug 10 03:42:43 dist-test-slave-ss4n krb5kdc[4049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333363, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.125@KRBTEST.COM for kudu/127.21.175.124@KRBTEST.COM
Aug 10 03:42:43 dist-test-slave-ss4n krb5kdc[4049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333363, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.124@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:42:43.833870 4320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.816824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.21.175.124:40187)
0810 03:42:43.817353 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:43.817369 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:43.817405 (+ 36us) server_negotiation.cc:407] Connection header received
0810 03:42:43.818182 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:43.818201 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:43.818536 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:43.818828 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:43.820010 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.821236 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.822542 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.823142 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.823324 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.826341 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:43.826347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:43.826360 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:43.826392 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:43.829852 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.830391 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.830398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.830405 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.830541 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.830895 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.830899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.830901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.831585 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:43.831841 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:43.832218 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 03:42:43.832663 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"spinlock_wait_cycles":768,"thread_start_us":158,"threads_started":1}
I20260810 03:42:43.836022 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.823636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:40187 (local address 127.0.0.1:39686)
0810 03:42:43.823866 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.823885 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:43.823984 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:43.825288 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:43.825291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:43.825492 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:43.825656 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.825663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.826823 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.826826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.827679 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.827690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.828423 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.828429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.828538 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.829018 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:43.829039 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:43.830715 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:43.834156 (+ 3441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.834160 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.834162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.834459 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.834821 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.834825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.834827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.834929 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.835417 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:43.835423 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:43.835536 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 03:42:43.835660 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 03:42:43.835826 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260810 03:42:43.836038 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.823687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39686 (local address 127.21.175.124:40187)
0810 03:42:43.824999 (+ 1312us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:43.825006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:43.825017 (+ 11us) server_negotiation.cc:407] Connection header received
0810 03:42:43.825047 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:43.825050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:43.825121 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:43.825772 (+ 651us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:43.825901 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.826726 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.827861 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.828269 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.828386 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.830876 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:43.830880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:43.830883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:43.830907 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:43.834002 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.834595 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.834598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.834600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.834704 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.835058 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.835061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.835063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.835294 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:43.835408 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:43.835717 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 03:42:43.835836 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1237,"spinlock_wait_cycles":2304,"thread_start_us":186,"threads_started":1}
I20260810 03:42:43.836894 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.825807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.21.175.126:44713)
0810 03:42:43.825963 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:43.825967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:43.825980 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:42:43.826026 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:43.826029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:43.826069 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:43.826179 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:43.827104 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.827844 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.829126 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.829465 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.829633 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.832451 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:43.832458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:43.832462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:43.832492 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:43.834181 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.834703 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.834706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.834708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.834799 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.835091 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.835095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.835097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.835327 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:43.835424 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:43.835825 (+ 401us) server_negotiation.cc:299] Negotiation successful
0810 03:42:43.835955 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 03:42:43.837853 4322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.824920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:44713 (local address 127.0.0.1:60074)
0810 03:42:43.825347 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.825403 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:43.825547 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:43.826239 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:43.826245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:43.826692 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:43.826898 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.826910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.827974 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.827978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.828967 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.828980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.829643 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.829647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.829791 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.830472 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:43.830489 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:43.832273 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:43.834312 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.834320 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.834325 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.834566 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.834898 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.834902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.834904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.834988 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.835476 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:43.835482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:43.835597 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 03:42:43.837106 (+ 1509us) client_negotiation.cc:240] Negotiation successful
0810 03:42:43.837309 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":6,"spinlock_wait_cycles":1792,"thread_start_us":133,"threads_started":1}
I20260810 03:42:43.839201 4317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:42:43.840320 4235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d: Bootstrap starting.
I20260810 03:42:43.843340 4235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d: Neither blocks nor log segments found. Creating new log.
Aug 10 03:42:43 dist-test-slave-ss4n krb5kdc[4049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:42:43.844437 4235 log.cc:824] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:42:43 dist-test-slave-ss4n krb5kdc[4049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333363, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.124@KRBTEST.COM for kudu/127.21.175.125@KRBTEST.COM
I20260810 03:42:43.846977 4235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d: No bootstrap required, opened a new log
I20260810 03:42:43.849311 4322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.839710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:44799 (local address 127.0.0.1:49278)
0810 03:42:43.839888 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.839902 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:43.839994 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:43.840239 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:43.840242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:43.840411 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:43.840519 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.840525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.841971 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.841974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.842926 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.842941 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.843710 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.843714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.843823 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.844353 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:43.844367 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:43.845878 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:43.847709 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.847715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.847717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.847921 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.848215 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.848219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.848221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.848312 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.848742 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:43.848745 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:43.848850 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 03:42:43.848960 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 03:42:43.849139 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260810 03:42:43.849311 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.839792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.21.175.125:44799)
0810 03:42:43.839907 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:43.839912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:43.839991 (+ 79us) server_negotiation.cc:407] Connection header received
0810 03:42:43.840070 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:43.840073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:43.840119 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:43.840209 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:43.840643 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.841458 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.843074 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.843441 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.843558 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.846026 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:43.846030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:43.846033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:43.846056 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:43.847565 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.848023 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.848027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.848029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.848120 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.848405 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.848410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.848412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.848639 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:43.848752 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:43.849031 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 03:42:43.849146 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 03:42:43.850261 4317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:42:43.851166 4235 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } }
I20260810 03:42:43.851469 4235 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:42:43.851554 4235 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf4c34f3e57540ddad79abb54729423d, State: Initialized, Role: FOLLOWER
I20260810 03:42:43.852106 4235 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } }
I20260810 03:42:43.852617 4324 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } } }
I20260810 03:42:43.852763 4324 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:42:43.854815 4235 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 03:42:43 dist-test-slave-ss4n krb5kdc[4049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333363, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.124@KRBTEST.COM for kudu/127.21.175.124@KRBTEST.COM
I20260810 03:42:43.856767 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.837709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60088 (local address 127.21.175.126:44713)
0810 03:42:43.837875 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:43.837879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:43.837895 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:42:43.837934 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:43.837940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:43.837991 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:43.838068 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:43.838965 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.839611 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.840804 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.841175 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.841285 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.844229 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:43.844233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:43.844236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:43.844268 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:43.854483 (+ 10215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.855138 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.855145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.855149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.855258 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.855568 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.855572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.855574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.855846 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:43.856118 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:43.856457 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 03:42:43.856601 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:42:43.860805 4322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.850831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:40187 (local address 127.0.0.1:39694)
0810 03:42:43.850938 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.850952 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:43.851029 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:43.851234 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:43.851236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:43.851367 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:43.851439 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.851443 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.852543 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.852547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.853348 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.853359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.854040 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.854043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.854123 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.854693 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:43.854708 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:43.856380 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:43.859156 (+ 2776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.859159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.859161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.859432 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.859728 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.859731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.859733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.859807 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.860301 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:43.860305 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:43.860394 (+ 89us) client_negotiation.cc:769] Sending connection context
0810 03:42:43.860474 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 03:42:43.860618 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 03:42:43.860797 4337 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:42:43.860868 4337 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:42:43.861753 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.850884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.21.175.124:40187)
0810 03:42:43.851014 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:43.851019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:43.851035 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:42:43.851086 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:43.851089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:43.851133 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:43.851234 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:43.851579 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.852414 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.853494 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.853920 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.854052 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.856612 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:43.856616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:43.856623 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:43.856658 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:43.859025 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.859552 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.859555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.859557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.859635 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.859902 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.859905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.859907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.860189 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:43.861282 (+ 1093us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:43.861379 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 03:42:43.861548 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:42:43.859326 4155 master_runner.cc:186] Error getting master registration for 127.21.175.126:44713: OK, instance_id { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" instance_seqno: 1786333363322901 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:42:43.864204 4155 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 10 03:42:43 dist-test-slave-ss4n krb5kdc[4049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.125@KRBTEST.COM
I20260810 03:42:43.865178 4317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517: Bootstrap starting.
I20260810 03:42:43.868238 4317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517: Neither blocks nor log segments found. Creating new log.
I20260810 03:42:43.869069 4317 log.cc:824] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517: Log is configured to *not* fsync() on all Append() calls
I20260810 03:42:43.869107 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.858340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.21.175.125:44799)
0810 03:42:43.858449 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:43.858454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:43.858467 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:42:43.858570 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:43.858573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:43.858617 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:43.858680 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:43.859512 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.860240 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.861308 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.861621 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.861733 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.864909 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:43.864913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:43.864916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:43.864943 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:43.866706 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.867292 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.867295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.867297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.867390 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.867774 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.867777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.867779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.868011 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:43.868219 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:43.868787 (+ 568us) server_negotiation.cc:299] Negotiation successful
0810 03:42:43.868919 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 03:42:43.871371 4317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517: No bootstrap required, opened a new log
I20260810 03:42:43.872996 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:42:43.875783 4317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } }
I20260810 03:42:43.876102 4317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:42:43.876169 4317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac6546f7ea6475eb84ecf952395c517, State: Initialized, Role: FOLLOWER
I20260810 03:42:43.876695 4317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } }
I20260810 03:42:43.877249 4348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } } }
I20260810 03:42:43.877401 4348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:42:43.878046 4317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:42:43.881671 4242 master_runner.cc:186] Error getting master registration for 127.21.175.126:44713: OK, instance_id { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" instance_seqno: 1786333363322901 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:42:43.885221 4242 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 03:42:43.891654 4360 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:42:43.894120 4360 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:42:43.895045 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.875436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39700 (local address 127.21.175.124:40187)
0810 03:42:43.875547 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:43.875551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:43.882348 (+ 6797us) server_negotiation.cc:407] Connection header received
0810 03:42:43.882400 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:43.882404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:43.882450 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:43.882520 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:43.886247 (+ 3727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.886917 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.888145 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.888483 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.888567 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.891148 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:43.891152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:43.891155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:43.891181 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:43.892903 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.893426 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.893432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.893435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.893567 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.893932 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.893935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.893937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.894172 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:43.894305 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:43.894699 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 03:42:43.894844 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 03:42:43.897151 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.876286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49294 (local address 127.21.175.125:44799)
0810 03:42:43.876423 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:43.876431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:43.882614 (+ 6183us) server_negotiation.cc:407] Connection header received
0810 03:42:43.882790 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:43.882794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:43.882844 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:43.882901 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:43.883838 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.884639 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.885809 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.886147 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.886228 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.888249 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:43.888253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:43.888256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:43.888283 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:43.890189 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.895633 (+ 5444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.895637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.895639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.895747 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.896079 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.896082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.896084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.896310 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:43.896424 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:43.896809 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 03:42:43.896952 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:42:43.900324 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.876208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60096 (local address 127.21.175.126:44713)
0810 03:42:43.876411 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:43.876415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:43.881872 (+ 5457us) server_negotiation.cc:407] Connection header received
0810 03:42:43.882089 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:43.882093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:43.882162 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:43.882238 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:43.885270 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.886094 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.891871 (+ 5777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.892263 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.892352 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.894381 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:43.894385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:43.894387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:43.894416 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:43.897759 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.898976 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.898980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.898981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.899076 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.899392 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.899395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.899397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.899598 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:43.899742 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:43.899999 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 03:42:43.900134 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260810 03:42:43.974843 4145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } attempt: 4
Aug 10 03:42:43 dist-test-slave-ss4n krb5kdc[4049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333363, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for kudu/127.21.175.124@KRBTEST.COM
I20260810 03:42:43.985395 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.975390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:40187 (local address 127.0.0.1:39716)
0810 03:42:43.975587 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:43.975603 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:43.975681 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:43.975973 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:43.975976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:43.976217 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:43.976338 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.976345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.977412 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.977415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.978210 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:43.978224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.979123 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.979127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:43.979240 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.979881 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:43.979897 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:43.981659 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:43.983742 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.983746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.983748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.984000 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.984303 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:43.984306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:43.984308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:43.984405 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:43.984846 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:43.984849 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:43.984958 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 03:42:43.985068 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 03:42:43.985219 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260810 03:42:43.985457 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:43.975489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.21.175.124:40187)
0810 03:42:43.975617 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:43.975622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:43.975664 (+ 42us) server_negotiation.cc:407] Connection header received
0810 03:42:43.975787 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:43.975790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:43.975842 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:43.975926 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:43.976474 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.977276 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.978358 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:43.978735 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:43.978839 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:43.981791 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:43.981796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:43.981799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:43.981828 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:43.983618 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.984105 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.984108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.984111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.984201 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:43.984500 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:43.984503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:43.984505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:43.984739 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:43.984845 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:43.985148 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 03:42:43.985281 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:42:43.989236 4145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4: Bootstrap starting.
I20260810 03:42:43.992281 4145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4: Neither blocks nor log segments found. Creating new log.
I20260810 03:42:43.993294 4145 log.cc:824] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4: Log is configured to *not* fsync() on all Append() calls
I20260810 03:42:43.995775 4145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4: No bootstrap required, opened a new log
I20260810 03:42:44.000881 4145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } }
I20260810 03:42:44.001226 4145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:42:44.001329 4145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 541a6c8d7fa44b2684cacd4c615566e4, State: Initialized, Role: FOLLOWER
I20260810 03:42:44.001919 4145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } }
I20260810 03:42:44.002663 4363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } } }
I20260810 03:42:44.002838 4363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:42:44.004158 4145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 03:42:44.009852 4377 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:42:44.010026 4377 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:42:44.015962 4065 master_runner.cc:263] No leader master found from master 541a6c8d7fa44b2684cacd4c615566e4
I20260810 03:42:44.016059 4065 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 03:42:44.056875 4266 tablet.cc:2369] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:42:44.064018 4180 tablet.cc:2369] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:42:44.067414 4088 tablet.cc:2369] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:42:44.101421 4324 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:42:44.101739 4324 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } }
I20260810 03:42:44.102957 4324 leader_election.cc:288] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 541a6c8d7fa44b2684cacd4c615566e4 (127.21.175.126:44713), 6ac6546f7ea6475eb84ecf952395c517 (127.21.175.124:40187)
I20260810 03:42:44.103593 4115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf4c34f3e57540ddad79abb54729423d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "541a6c8d7fa44b2684cacd4c615566e4" is_pre_election: true
I20260810 03:42:44.103660 4288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf4c34f3e57540ddad79abb54729423d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac6546f7ea6475eb84ecf952395c517" is_pre_election: true
I20260810 03:42:44.103986 4115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf4c34f3e57540ddad79abb54729423d in term 0.
I20260810 03:42:44.103994 4288 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf4c34f3e57540ddad79abb54729423d in term 0.
I20260810 03:42:44.104445 4176 leader_election.cc:302] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 541a6c8d7fa44b2684cacd4c615566e4, 6ac6546f7ea6475eb84ecf952395c517, bf4c34f3e57540ddad79abb54729423d; no voters:
I20260810 03:42:44.104959 4324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:42:44.105043 4324 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:42:44.105086 4324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:42:44.106421 4324 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } }
I20260810 03:42:44.106783 4324 leader_election.cc:288] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [CANDIDATE]: Term 1 election: Requested vote from peers 541a6c8d7fa44b2684cacd4c615566e4 (127.21.175.126:44713), 6ac6546f7ea6475eb84ecf952395c517 (127.21.175.124:40187)
I20260810 03:42:44.107097 4115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf4c34f3e57540ddad79abb54729423d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "541a6c8d7fa44b2684cacd4c615566e4"
I20260810 03:42:44.107117 4288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf4c34f3e57540ddad79abb54729423d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac6546f7ea6475eb84ecf952395c517"
I20260810 03:42:44.107240 4288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:42:44.107247 4115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:42:44.108294 4115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf4c34f3e57540ddad79abb54729423d in term 1.
I20260810 03:42:44.108448 4288 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf4c34f3e57540ddad79abb54729423d in term 1.
I20260810 03:42:44.108598 4173 leader_election.cc:302] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 541a6c8d7fa44b2684cacd4c615566e4, bf4c34f3e57540ddad79abb54729423d; no voters:
I20260810 03:42:44.108821 4324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:42:44.109032 4324 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [term 1 LEADER]: Becoming Leader. State: Replica: bf4c34f3e57540ddad79abb54729423d, State: Running, Role: LEADER
I20260810 03:42:44.109371 4324 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } }
I20260810 03:42:44.111505 4379 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf4c34f3e57540ddad79abb54729423d. Latest consensus state: current_term: 1 leader_uuid: "bf4c34f3e57540ddad79abb54729423d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } } }
I20260810 03:42:44.111690 4379 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [sys.catalog]: This master's current role is: LEADER
I20260810 03:42:44.112144 4381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:42:44.113221 4381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:42:44.117977 4115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4 [term 1 FOLLOWER]: Refusing update from remote peer bf4c34f3e57540ddad79abb54729423d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:42:44.117983 4288 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517 [term 1 FOLLOWER]: Refusing update from remote peer bf4c34f3e57540ddad79abb54729423d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:42:44.118700 4379 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [LEADER]: Connected to new peer: Peer: permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:42:44.118872 4324 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:42:44.122789 4348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf4c34f3e57540ddad79abb54729423d. Latest consensus state: current_term: 1 leader_uuid: "bf4c34f3e57540ddad79abb54729423d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } } }
I20260810 03:42:44.122813 4363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf4c34f3e57540ddad79abb54729423d. Latest consensus state: current_term: 1 leader_uuid: "bf4c34f3e57540ddad79abb54729423d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } } }
I20260810 03:42:44.122917 4348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:42:44.122932 4363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:42:44.125361 4382 mvcc.cc:205] Tried to move back new op lower bound from 7316821459420770304 to 7316821459394850816. Current Snapshot: MvccSnapshot[applied={T|T < 7316821459420770304}]
I20260810 03:42:44.125365 4379 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf4c34f3e57540ddad79abb54729423d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } } }
I20260810 03:42:44.125541 4379 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [sys.catalog]: This master's current role is: LEADER
I20260810 03:42:44.125813 4348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf4c34f3e57540ddad79abb54729423d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } } }
I20260810 03:42:44.125931 4348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:42:44.125900 4383 mvcc.cc:205] Tried to move back new op lower bound from 7316821459420770304 to 7316821459394850816. Current Snapshot: MvccSnapshot[applied={T|T < 7316821459420770304}]
I20260810 03:42:44.128258 4381 catalog_manager.cc:1347] Generated new cluster ID: 36b84f599e0547019f761bc5d1a2a099
I20260810 03:42:44.128328 4381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:42:44.131625 4379 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf4c34f3e57540ddad79abb54729423d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } } }
I20260810 03:42:44.131742 4379 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d [sys.catalog]: This master's current role is: LEADER
I20260810 03:42:44.132539 4363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf4c34f3e57540ddad79abb54729423d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "541a6c8d7fa44b2684cacd4c615566e4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44713 } } peers { permanent_uuid: "bf4c34f3e57540ddad79abb54729423d" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 44799 } } peers { permanent_uuid: "6ac6546f7ea6475eb84ecf952395c517" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 40187 } } }
I20260810 03:42:44.132656 4363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:42:44.157023 4381 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:42:44.158363 4381 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:42:44.168772 4381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf4c34f3e57540ddad79abb54729423d: Generated new TSK 0
I20260810 03:42:44.169709 4381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:42:44.895576 4360 catalog_manager.cc:1259] Loaded cluster ID: 36b84f599e0547019f761bc5d1a2a099
I20260810 03:42:44.895663 4360 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517: loading cluster ID for follower catalog manager: success
I20260810 03:42:44.897361 4360 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517: acquiring CA information for follower catalog manager: success
I20260810 03:42:44.898589 4360 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6ac6546f7ea6475eb84ecf952395c517: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:42:45.011533 4377 catalog_manager.cc:1259] Loaded cluster ID: 36b84f599e0547019f761bc5d1a2a099
I20260810 03:42:45.011665 4377 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4: loading cluster ID for follower catalog manager: success
I20260810 03:42:45.013442 4377 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4: acquiring CA information for follower catalog manager: success
I20260810 03:42:45.014731 4377 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 541a6c8d7fa44b2684cacd4c615566e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:42:45.229525 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 4065
I20260810 03:42:45.238219 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 4155
I20260810 03:42:45.246300 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 4242
2026-08-10T03:42:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3445 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:42:45 dist-test-slave-ss4n krb5kdc[4397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:42:45 dist-test-slave-ss4n krb5kdc[4397](info): set up 2 sockets
Aug 10 03:42:45 dist-test-slave-ss4n krb5kdc[4397](info): commencing operation
krb5kdc: starting...
W20260810 03:42:46.401873 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:42:46 dist-test-slave-ss4n krb5kdc[4397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:42:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:42:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:42:46.545068 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:37825
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:46479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:37825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:42:46.662688 4418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:46.662987 4418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:46.663061 4418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:46.667192 4418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:42:46.667260 4418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:46.667281 4418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:46.667297 4418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:42:46.667313 4418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:42:46.672987 4418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:46479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:37825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:37825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:46.674206 4418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:46.675271 4418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:46.684841 4425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:46.684849 4428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:46.684880 4426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:46.685165 4418 server_base.cc:1034] running on GCE node
I20260810 03:42:46.685899 4418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:46.687086 4418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:46.688299 4418 hybrid_clock.cc:648] HybridClock initialized: now 1786333366688257 us; error 54 us; skew 500 ppm
Aug 10 03:42:46 dist-test-slave-ss4n krb5kdc[4397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333366, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:46.691619 4418 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:42:46.692721 4418 webserver.cc:460] Webserver started at http://127.21.175.126:46851/ using document root <none> and password file <none>
I20260810 03:42:46.693347 4418 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:46.693451 4418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:46.693738 4418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:42:46.696034 4418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "5cc5f0b34b7e408c97349afb69fd4fb9"
format_stamp: "Formatted at 2026-08-10 03:42:46 on dist-test-slave-ss4n"
server_key: "7c452a66d57f143b4f3a7f2180360090"
server_key_iv: "2322e2c8b8ccccb249d937bc6b3673e4"
server_key_version: "encryptionkey@0"
I20260810 03:42:46.696667 4418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "5cc5f0b34b7e408c97349afb69fd4fb9"
format_stamp: "Formatted at 2026-08-10 03:42:46 on dist-test-slave-ss4n"
server_key: "7c452a66d57f143b4f3a7f2180360090"
server_key_iv: "2322e2c8b8ccccb249d937bc6b3673e4"
server_key_version: "encryptionkey@0"
I20260810 03:42:46.700865 4418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 03:42:46.703621 4435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:46.705004 4418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:42:46.705188 4418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "5cc5f0b34b7e408c97349afb69fd4fb9"
format_stamp: "Formatted at 2026-08-10 03:42:46 on dist-test-slave-ss4n"
server_key: "7c452a66d57f143b4f3a7f2180360090"
server_key_iv: "2322e2c8b8ccccb249d937bc6b3673e4"
server_key_version: "encryptionkey@0"
I20260810 03:42:46.705328 4418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:46.723048 4418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:46.726881 4418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:46.727135 4418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:46.734908 4418 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:37825
I20260810 03:42:46.734973 4495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:37825 every 8 connection(s)
I20260810 03:42:46.735994 4418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:42:46.739202 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:42:46.742848 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 4418
I20260810 03:42:46.742990 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:42:46.743279 22205 external_mini_cluster.cc:1428] Setting key 566f004cff553e116510550baa1c2aba
I20260810 03:42:46.745862 4496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9: Bootstrap starting.
I20260810 03:42:46.748261 4496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9: Neither blocks nor log segments found. Creating new log.
I20260810 03:42:46.749081 4496 log.cc:824] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9: Log is configured to *not* fsync() on all Append() calls
I20260810 03:42:46.751174 4496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9: No bootstrap required, opened a new log
Aug 10 03:42:46 dist-test-slave-ss4n krb5kdc[4397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:42:46.754904 4496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc5f0b34b7e408c97349afb69fd4fb9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37825 } }
I20260810 03:42:46.755206 4496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:42:46.755312 4496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc5f0b34b7e408c97349afb69fd4fb9, State: Initialized, Role: FOLLOWER
I20260810 03:42:46.755872 4496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc5f0b34b7e408c97349afb69fd4fb9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37825 } }
I20260810 03:42:46.756040 4496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:42:46.756138 4496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:42:46.756266 4496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:42:46.757268 4496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc5f0b34b7e408c97349afb69fd4fb9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37825 } }
I20260810 03:42:46.757694 4496 leader_election.cc:302] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cc5f0b34b7e408c97349afb69fd4fb9; no voters:
I20260810 03:42:46.758102 4496 leader_election.cc:288] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:42:46.758276 4502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:42:46.758492 4502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [term 1 LEADER]: Becoming Leader. State: Replica: 5cc5f0b34b7e408c97349afb69fd4fb9, State: Running, Role: LEADER
I20260810 03:42:46.758834 4502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc5f0b34b7e408c97349afb69fd4fb9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37825 } }
I20260810 03:42:46.759305 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:46.744763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51216 (local address 127.21.175.126:37825)
0810 03:42:46.745280 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:46.745297 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:46.745337 (+ 40us) server_negotiation.cc:407] Connection header received
0810 03:42:46.746119 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:46.746138 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:46.746508 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:46.746868 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:46.747739 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:46.748828 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:46.749846 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:46.750343 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:46.750520 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:46.753191 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:46.753196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:46.753210 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:46.753239 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:46.755839 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:46.756409 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:46.756422 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:46.756427 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:46.756549 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:46.756873 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:46.756877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:46.756878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:46.757333 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:46.757597 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:46.757916 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 03:42:46.758270 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":169,"threads_started":1}
I20260810 03:42:46.759609 4496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:42:46.760108 4502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cc5f0b34b7e408c97349afb69fd4fb9. Latest consensus state: current_term: 1 leader_uuid: "5cc5f0b34b7e408c97349afb69fd4fb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc5f0b34b7e408c97349afb69fd4fb9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37825 } } }
I20260810 03:42:46.760221 4502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [sys.catalog]: This master's current role is: LEADER
I20260810 03:42:46.760574 4504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cc5f0b34b7e408c97349afb69fd4fb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc5f0b34b7e408c97349afb69fd4fb9" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37825 } } }
I20260810 03:42:46.760692 4504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9 [sys.catalog]: This master's current role is: LEADER
I20260810 03:42:46.762593 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 03:42:46.765372 4517 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:42:46.765446 4517 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:42:46.765564 4506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:42:46.766357 4506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:42:46.772070 4506 catalog_manager.cc:1347] Generated new cluster ID: 3a9538b22d3b47e5934a9e448b0e1185
I20260810 03:42:46.772154 4506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:42:46.779902 4506 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:42:46.781023 4506 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:42:46.797276 4506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cc5f0b34b7e408c97349afb69fd4fb9: Generated new TSK 0
I20260810 03:42:46.798115 4506 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:42:46.819185 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 4418
2026-08-10T03:42:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1571 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:42:46 dist-test-slave-ss4n krb5kdc[4526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:42:46 dist-test-slave-ss4n krb5kdc[4526](info): set up 2 sockets
Aug 10 03:42:46 dist-test-slave-ss4n krb5kdc[4526](info): commencing operation
krb5kdc: starting...
W20260810 03:42:48.012704 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:42:48 dist-test-slave-ss4n krb5kdc[4526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:42:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:42:48.151106 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:36335
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:36335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:42:48.266165 4541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:48.266503 4541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:48.266600 4541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:48.270694 4541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:42:48.270789 4541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:48.270829 4541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:48.270887 4541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:42:48.270926 4541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:42:48.276373 4541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:36335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:36335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:48.277629 4541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:48.278712 4541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:48.287715 4546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:48.287752 4547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:48.287739 4549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:48.288183 4541 server_base.cc:1034] running on GCE node
I20260810 03:42:48.288805 4541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:48.290225 4541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:48.291420 4541 hybrid_clock.cc:648] HybridClock initialized: now 1786333368291426 us; error 50 us; skew 500 ppm
Aug 10 03:42:48 dist-test-slave-ss4n krb5kdc[4526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333368, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:48.294539 4541 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:42:48.295554 4541 webserver.cc:460] Webserver started at http://127.21.175.126:38097/ using document root <none> and password file <none>
I20260810 03:42:48.296180 4541 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:48.296280 4541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:48.296522 4541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:42:48.298700 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "4de9004f7e474d2db46d3fa2e8170734"
format_stamp: "Formatted at 2026-08-10 03:42:48 on dist-test-slave-ss4n"
server_key: "bf9f7c3463406d3783f9cb5b8289f152"
server_key_iv: "b07a16796e47e4ed767945e9bd3f16e6"
server_key_version: "encryptionkey@0"
I20260810 03:42:48.299293 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "4de9004f7e474d2db46d3fa2e8170734"
format_stamp: "Formatted at 2026-08-10 03:42:48 on dist-test-slave-ss4n"
server_key: "bf9f7c3463406d3783f9cb5b8289f152"
server_key_iv: "b07a16796e47e4ed767945e9bd3f16e6"
server_key_version: "encryptionkey@0"
I20260810 03:42:48.303411 4541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 03:42:48.306315 4558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:48.307590 4541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:42:48.307737 4541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "4de9004f7e474d2db46d3fa2e8170734"
format_stamp: "Formatted at 2026-08-10 03:42:48 on dist-test-slave-ss4n"
server_key: "bf9f7c3463406d3783f9cb5b8289f152"
server_key_iv: "b07a16796e47e4ed767945e9bd3f16e6"
server_key_version: "encryptionkey@0"
I20260810 03:42:48.307864 4541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:48.331810 4541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:48.335422 4541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:48.335654 4541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:48.343608 4541 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:36335
I20260810 03:42:48.343637 4617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:36335 every 8 connection(s)
I20260810 03:42:48.344645 4541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:42:48.347645 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:42:48.349242 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 4541
I20260810 03:42:48.349372 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:42:48.349720 22205 external_mini_cluster.cc:1428] Setting key 95b5561e496a471da9d3e171a8a3db78
I20260810 03:42:48.354238 4619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734: Bootstrap starting.
I20260810 03:42:48.356599 4619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734: Neither blocks nor log segments found. Creating new log.
I20260810 03:42:48.357548 4619 log.cc:824] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:42:48 dist-test-slave-ss4n krb5kdc[4526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:42:48.359876 4619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734: No bootstrap required, opened a new log
I20260810 03:42:48.363744 4619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } }
I20260810 03:42:48.364066 4619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:42:48.364148 4619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de9004f7e474d2db46d3fa2e8170734, State: Initialized, Role: FOLLOWER
I20260810 03:42:48.364650 4619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } }
I20260810 03:42:48.364774 4619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:42:48.364859 4619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:42:48.364872 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:48.351301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.21.175.126:36335)
0810 03:42:48.351844 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:48.351859 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:48.351895 (+ 36us) server_negotiation.cc:407] Connection header received
0810 03:42:48.352532 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:48.352557 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:48.352892 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:48.353229 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:48.354124 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:48.355143 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:48.356016 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:48.356501 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:48.356685 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:48.359783 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:48.359789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:48.359802 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:48.359830 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:48.361768 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:48.362270 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:48.362276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:48.362280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:48.362394 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:48.362741 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:48.362745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:48.362747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:48.363165 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:48.363383 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:48.363679 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 03:42:48.363977 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":153,"threads_started":1}
I20260810 03:42:48.364984 4619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:42:48.366173 4619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } }
I20260810 03:42:48.366528 4619 leader_election.cc:302] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4de9004f7e474d2db46d3fa2e8170734; no voters:
I20260810 03:42:48.366834 4619 leader_election.cc:288] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:42:48.366957 4624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:42:48.367179 4624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 1 LEADER]: Becoming Leader. State: Replica: 4de9004f7e474d2db46d3fa2e8170734, State: Running, Role: LEADER
I20260810 03:42:48.367547 4624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } }
I20260810 03:42:48.368131 4619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:42:48.368734 4626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4de9004f7e474d2db46d3fa2e8170734. Latest consensus state: current_term: 1 leader_uuid: "4de9004f7e474d2db46d3fa2e8170734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } } }
I20260810 03:42:48.368860 4626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [sys.catalog]: This master's current role is: LEADER
I20260810 03:42:48.369225 4625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4de9004f7e474d2db46d3fa2e8170734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } } }
I20260810 03:42:48.369329 4625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [sys.catalog]: This master's current role is: LEADER
I20260810 03:42:48.369622 4630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:42:48.372339 4630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:42:48.378611 4630 catalog_manager.cc:1347] Generated new cluster ID: a329cb1f97c446ba8b0a861c6e2e98f6
I20260810 03:42:48.378685 4630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:42:48.391844 4630 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:42:48.393165 4630 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:42:48.405814 4630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734: Generated new TSK 0
I20260810 03:42:48.406672 4630 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:42:48.445617 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36335
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:42:48.573770 4647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:48.574095 4647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:48.574179 4647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:48.578903 4647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:48.579007 4647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:48.579113 4647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:42:48.585071 4647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:48.586412 4647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:48.587560 4647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:48.597956 4655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:48.598140 4647 server_base.cc:1034] running on GCE node
W20260810 03:42:48.597936 4652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:48.597965 4653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:48.598702 4647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:48.599378 4647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:48.600564 4647 hybrid_clock.cc:648] HybridClock initialized: now 1786333368600546 us; error 31 us; skew 500 ppm
Aug 10 03:42:48 dist-test-slave-ss4n krb5kdc[4526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333368, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:48.603615 4647 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:42:48.604511 4647 webserver.cc:460] Webserver started at http://127.21.175.65:36477/ using document root <none> and password file <none>
I20260810 03:42:48.605070 4647 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:48.605161 4647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:48.605396 4647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:42:48.607477 4647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "4b6e6da672e44e2396fc704ec88bcfb3"
format_stamp: "Formatted at 2026-08-10 03:42:48 on dist-test-slave-ss4n"
server_key: "df02fffb25683b22cd43fb159702ac60"
server_key_iv: "9b00ce6b16e604266f0d69bb2f648614"
server_key_version: "encryptionkey@0"
I20260810 03:42:48.608070 4647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "4b6e6da672e44e2396fc704ec88bcfb3"
format_stamp: "Formatted at 2026-08-10 03:42:48 on dist-test-slave-ss4n"
server_key: "df02fffb25683b22cd43fb159702ac60"
server_key_iv: "9b00ce6b16e604266f0d69bb2f648614"
server_key_version: "encryptionkey@0"
I20260810 03:42:48.612010 4647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:42:48.614609 4662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:48.615842 4647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:42:48.616024 4647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "4b6e6da672e44e2396fc704ec88bcfb3"
format_stamp: "Formatted at 2026-08-10 03:42:48 on dist-test-slave-ss4n"
server_key: "df02fffb25683b22cd43fb159702ac60"
server_key_iv: "9b00ce6b16e604266f0d69bb2f648614"
server_key_version: "encryptionkey@0"
I20260810 03:42:48.616143 4647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:48.627879 4647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:48.631349 4647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:48.631577 4647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:48.632220 4647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:42:48.633143 4647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:42:48.633203 4647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:48.633253 4647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:42:48.633280 4647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:48.642414 4647 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:35749
I20260810 03:42:48.642513 4798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:35749 every 8 connection(s)
I20260810 03:42:48.643383 4647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:42:48.643977 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 4647
I20260810 03:42:48.644131 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:42:48.644522 22205 external_mini_cluster.cc:1428] Setting key f528d5d10f421108e769d13fbd28864a
Aug 10 03:42:48 dist-test-slave-ss4n krb5kdc[4526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333368, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
I20260810 03:42:48.658962 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:48.645636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:50897 (local address 127.21.175.126:36335)
0810 03:42:48.645792 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:48.645797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:48.646875 (+ 1078us) server_negotiation.cc:407] Connection header received
0810 03:42:48.648374 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:48.648378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:48.648429 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:48.648514 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:48.649803 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:48.650355 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:48.651553 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:48.651797 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:48.651909 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:48.654738 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:48.654742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:48.654744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:48.654769 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:48.656632 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:48.657180 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:48.657184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:48.657188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:48.657289 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:48.657752 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:48.657755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:48.657757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:48.658022 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:48.658122 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:48.658650 (+ 528us) server_negotiation.cc:299] Negotiation successful
0810 03:42:48.658785 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:42:48.659778 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:48.646120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:50897)
0810 03:42:48.646709 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:48.646762 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:48.648166 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:48.648707 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:48.648714 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:48.649084 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:48.649653 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:48.649670 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:48.650496 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:48.650503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:48.651386 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:48.651397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:48.651953 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:48.651957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:48.652142 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:48.652762 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:48.652779 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:48.654559 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:48.656761 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:48.656766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:48.656783 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:48.657062 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:48.657389 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:48.657392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:48.657394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:48.657640 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:48.658122 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:48.658130 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:48.658401 (+ 271us) client_negotiation.cc:769] Sending connection context
0810 03:42:48.658597 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 03:42:48.658940 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":184,"threads_started":1}
I20260810 03:42:48.661201 4799 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36335
I20260810 03:42:48.661473 4799 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:48.662106 4799 heartbeater.cc:511] Master 127.21.175.126:36335 requested a full tablet report, sending...
I20260810 03:42:48.663960 4577 ts_manager.cc:194] Registered new tserver with Master: 4b6e6da672e44e2396fc704ec88bcfb3 (127.21.175.65:35749)
I20260810 03:42:48.665261 4577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:50897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:42:48.677958 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36335
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:42:48.803351 4804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:48.803614 4804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:48.803658 4804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:48.807897 4804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:48.807961 4804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:48.808019 4804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:42:48.813282 4804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:48.814358 4804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:48.815402 4804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:48.825654 4811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:48.825695 4810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:48.825686 4804 server_base.cc:1034] running on GCE node
W20260810 03:42:48.825711 4813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:48.826349 4804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:48.827008 4804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:48.828176 4804 hybrid_clock.cc:648] HybridClock initialized: now 1786333368828139 us; error 49 us; skew 500 ppm
Aug 10 03:42:48 dist-test-slave-ss4n krb5kdc[4526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333368, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:48.831099 4804 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:42:48.832072 4804 webserver.cc:460] Webserver started at http://127.21.175.66:41145/ using document root <none> and password file <none>
I20260810 03:42:48.832727 4804 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:48.832829 4804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:48.833086 4804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:42:48.835233 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "046f21098e2344e3b444ad86670ab346"
format_stamp: "Formatted at 2026-08-10 03:42:48 on dist-test-slave-ss4n"
server_key: "9d38b2b572fd77037a811de2822de28a"
server_key_iv: "e02ecf8d3ce8b28d82e48a72950e7c15"
server_key_version: "encryptionkey@0"
I20260810 03:42:48.835807 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "046f21098e2344e3b444ad86670ab346"
format_stamp: "Formatted at 2026-08-10 03:42:48 on dist-test-slave-ss4n"
server_key: "9d38b2b572fd77037a811de2822de28a"
server_key_iv: "e02ecf8d3ce8b28d82e48a72950e7c15"
server_key_version: "encryptionkey@0"
I20260810 03:42:48.839640 4804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 03:42:48.842116 4820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:48.843324 4804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:42:48.843458 4804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "046f21098e2344e3b444ad86670ab346"
format_stamp: "Formatted at 2026-08-10 03:42:48 on dist-test-slave-ss4n"
server_key: "9d38b2b572fd77037a811de2822de28a"
server_key_iv: "e02ecf8d3ce8b28d82e48a72950e7c15"
server_key_version: "encryptionkey@0"
I20260810 03:42:48.843572 4804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:48.857388 4804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:48.860522 4804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:48.860736 4804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:48.861368 4804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:42:48.862273 4804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:42:48.862347 4804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:48.862423 4804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:42:48.862468 4804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:48.872008 4804 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:35821
I20260810 03:42:48.872021 4948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:35821 every 8 connection(s)
I20260810 03:42:48.872975 4804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:42:48.876205 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 4804
I20260810 03:42:48.876322 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:42:48.876588 22205 external_mini_cluster.cc:1428] Setting key b712989f58d75d2950ab37c8a807c8a0
Aug 10 03:42:48 dist-test-slave-ss4n krb5kdc[4526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333368, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
I20260810 03:42:48.888098 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:48.874883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:34177 (local address 127.21.175.126:36335)
0810 03:42:48.875073 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:48.875079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:48.876972 (+ 1893us) server_negotiation.cc:407] Connection header received
0810 03:42:48.877468 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:48.877471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:48.877613 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:48.877684 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:48.878886 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:48.879417 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:48.881050 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:48.881273 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:48.881359 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:48.884344 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:48.884347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:48.884349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:48.884375 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:48.885927 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:48.886467 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:48.886472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:48.886474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:48.886564 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:48.886916 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:48.886921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:48.886923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:48.887123 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:48.887232 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:48.887788 (+ 556us) server_negotiation.cc:299] Negotiation successful
0810 03:42:48.887919 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 03:42:48.888975 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:48.875331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:34177)
0810 03:42:48.875829 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:48.875871 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:48.877236 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:48.877897 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:48.877904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:48.878257 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:48.878742 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:48.878755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:48.879557 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:48.879560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:48.880914 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:48.880926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:48.881434 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:48.881440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:48.881720 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:48.882398 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:48.882425 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:48.884159 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:48.886053 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:48.886060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:48.886077 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:48.886357 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:48.886663 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:48.886666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:48.886669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:48.886818 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:48.887232 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:48.887241 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:48.887535 (+ 294us) client_negotiation.cc:769] Sending connection context
0810 03:42:48.887743 (+ 208us) client_negotiation.cc:240] Negotiation successful
0810 03:42:48.888086 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":147,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
I20260810 03:42:48.890317 4949 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36335
I20260810 03:42:48.890619 4949 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:48.891196 4949 heartbeater.cc:511] Master 127.21.175.126:36335 requested a full tablet report, sending...
I20260810 03:42:48.892328 4577 ts_manager.cc:194] Registered new tserver with Master: 046f21098e2344e3b444ad86670ab346 (127.21.175.66:35821)
I20260810 03:42:48.893025 4577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:34177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:42:48.909349 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36335
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:42:49.024349 4956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:49.024637 4956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:49.024682 4956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:49.029212 4956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:49.029271 4956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:49.029332 4956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:42:49.034847 4956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:49.035928 4956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:49.036937 4956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:49.047258 4963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:49.047225 4964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:49.047230 4966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:49.048074 4956 server_base.cc:1034] running on GCE node
I20260810 03:42:49.048448 4956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:49.049062 4956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:49.050256 4956 hybrid_clock.cc:648] HybridClock initialized: now 1786333369050216 us; error 51 us; skew 500 ppm
Aug 10 03:42:49 dist-test-slave-ss4n krb5kdc[4526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:49.053210 4956 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:42:49.054198 4956 webserver.cc:460] Webserver started at http://127.21.175.67:36155/ using document root <none> and password file <none>
I20260810 03:42:49.054782 4956 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:49.054878 4956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:49.055122 4956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:42:49.057232 4956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "27dd45a0f7d9402aa766cc714ef0c60d"
format_stamp: "Formatted at 2026-08-10 03:42:49 on dist-test-slave-ss4n"
server_key: "302fb5c905eeb5962c7ce28c33107ebf"
server_key_iv: "f018eb3c40744f81bb5178512541a3a2"
server_key_version: "encryptionkey@0"
I20260810 03:42:49.057916 4956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "27dd45a0f7d9402aa766cc714ef0c60d"
format_stamp: "Formatted at 2026-08-10 03:42:49 on dist-test-slave-ss4n"
server_key: "302fb5c905eeb5962c7ce28c33107ebf"
server_key_iv: "f018eb3c40744f81bb5178512541a3a2"
server_key_version: "encryptionkey@0"
I20260810 03:42:49.061846 4956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:42:49.064496 4975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:49.065924 4956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:42:49.066084 4956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "27dd45a0f7d9402aa766cc714ef0c60d"
format_stamp: "Formatted at 2026-08-10 03:42:49 on dist-test-slave-ss4n"
server_key: "302fb5c905eeb5962c7ce28c33107ebf"
server_key_iv: "f018eb3c40744f81bb5178512541a3a2"
server_key_version: "encryptionkey@0"
I20260810 03:42:49.066210 4956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:49.103272 4956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:49.106868 4956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:49.107116 4956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:49.107752 4956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:42:49.108588 4956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:42:49.108669 4956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:49.108748 4956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:42:49.108804 4956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:49.118264 4956 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:40255
I20260810 03:42:49.118310 5111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:40255 every 8 connection(s)
I20260810 03:42:49.119207 4956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:42:49.127488 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 4956
I20260810 03:42:49.127604 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:42:49.128023 22205 external_mini_cluster.cc:1428] Setting key 1a059fe32fc49fbc0656c8a6193a5495
Aug 10 03:42:49 dist-test-slave-ss4n krb5kdc[4526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:42:49.133844 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.121158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:38963 (local address 127.21.175.126:36335)
0810 03:42:49.121309 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:49.121312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:49.122223 (+ 911us) server_negotiation.cc:407] Connection header received
0810 03:42:49.123738 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:49.123742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:49.123803 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:49.123910 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:49.125101 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.125752 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.126907 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.127140 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.127258 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.130325 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:49.130329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:49.130332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:49.130361 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:49.131738 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:49.132270 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:49.132275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:49.132277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:49.132363 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:49.132725 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:49.132730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:49.132731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:49.132924 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:49.133043 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:49.133550 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 03:42:49.133672 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:42:49.134723 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.121559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:38963)
0810 03:42:49.122027 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:49.122079 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:49.123464 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:49.124097 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:49.124104 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:49.124462 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:49.124947 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:49.124960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.125882 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.125890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:49.126776 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:49.126785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.127298 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.127303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:49.127475 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.128268 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:49.128285 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:49.130161 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:49.131861 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:49.131868 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:49.131883 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:49.132162 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:49.132469 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:49.132472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:49.132474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:49.132626 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:49.133034 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:49.133041 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:49.133295 (+ 254us) client_negotiation.cc:769] Sending connection context
0810 03:42:49.133488 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 03:42:49.133846 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":119,"threads_started":1}
I20260810 03:42:49.135905 5112 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36335
I20260810 03:42:49.136178 5112 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:49.136693 5112 heartbeater.cc:511] Master 127.21.175.126:36335 requested a full tablet report, sending...
I20260810 03:42:49.137832 4577 ts_manager.cc:194] Registered new tserver with Master: 27dd45a0f7d9402aa766cc714ef0c60d (127.21.175.67:40255)
I20260810 03:42:49.138428 4577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:38963
I20260810 03:42:49.142480 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:42:49.152321 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.144762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43932 (local address 127.21.175.126:36335)
0810 03:42:49.144883 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:49.144886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:49.145108 (+ 222us) server_negotiation.cc:407] Connection header received
0810 03:42:49.145209 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:49.145212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:49.145250 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:49.145319 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:49.146132 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.146740 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.147656 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.147870 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.147949 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.149434 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:49.149437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:49.149439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:49.149459 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:49.150764 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:49.151180 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:49.151182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:49.151184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:49.151255 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:49.151485 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:49.151486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:49.151487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:49.151693 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:49.151796 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:49.152085 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 03:42:49.152199 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 03:42:49.155491 4577 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:42:49.158638 4577 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:42:49.174539 5128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.168209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.21.175.65:35749)
0810 03:42:49.168883 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:49.168889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:49.169192 (+ 303us) server_negotiation.cc:407] Connection header received
0810 03:42:49.169577 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:49.169585 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:49.169794 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:49.169949 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:42:49.170486 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.171371 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.172861 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.173556 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.173666 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.173817 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:49.174227 (+ 410us) server_negotiation.cc:299] Negotiation successful
0810 03:42:49.174345 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":128,"threads_started":1}
I20260810 03:42:49.175032 5124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.167896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:40255 (local address 127.0.0.1:33268)
0810 03:42:49.169051 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:49.169098 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:49.169555 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:49.170222 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:49.170226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:49.170250 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:42:49.170363 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:49.170370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.171790 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.171793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:49.173217 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:49.173229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.174394 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.174398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:49.174509 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.174533 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:42:49.174624 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 03:42:49.174873 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":174,"thread_start_us":88,"threads_started":1}
I20260810 03:42:49.175197 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.168064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33268 (local address 127.21.175.67:40255)
0810 03:42:49.168648 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:49.168661 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:49.169191 (+ 530us) server_negotiation.cc:407] Connection header received
0810 03:42:49.169707 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:49.169712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:49.169905 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:49.170057 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:42:49.170515 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.171655 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.173356 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.174227 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.174333 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.174461 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:49.174922 (+ 461us) server_negotiation.cc:299] Negotiation successful
0810 03:42:49.175042 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":162,"threads_started":1}
I20260810 03:42:49.175429 5130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.168895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.21.175.66:35821)
0810 03:42:49.169536 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:49.169548 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:49.169571 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:42:49.169637 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:49.169641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:49.169794 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:49.169958 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:42:49.170355 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.171596 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.173630 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.174261 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.174431 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.174593 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:49.175123 (+ 530us) server_negotiation.cc:299] Negotiation successful
0810 03:42:49.175249 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":118,"threads_started":1}
I20260810 03:42:49.175047 5125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.168712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:35821 (local address 127.0.0.1:49962)
0810 03:42:49.169324 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:49.169337 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:49.169412 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:49.169974 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:49.169983 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:49.170013 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:42:49.170164 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:49.170175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.171766 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.171770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:49.173217 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:49.173229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.174664 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.174670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:49.174767 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.174789 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 03:42:49.174865 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 03:42:49.174924 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":91,"thread_start_us":98,"threads_started":1}
I20260810 03:42:49.175737 5129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.167896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:35749 (local address 127.0.0.1:34980)
0810 03:42:49.169046 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:49.169098 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:49.169303 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:49.170202 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:49.170204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:49.170220 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:42:49.170353 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:49.170359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.171526 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.171532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:49.172732 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:49.172743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.173881 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.173885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:49.173997 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.174039 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 03:42:49.174728 (+ 689us) client_negotiation.cc:240] Negotiation successful
0810 03:42:49.175616 (+ 888us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"thread_start_us":108,"threads_started":1}
I20260810 03:42:49.178740 4875 tablet_service.cc:1467] Processing CreateTablet for tablet d01d02061df34778a2587e6e0908ee61 (DEFAULT_TABLE table=test-table [id=744ce97a353a43ab9985a235c2d3521e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:42:49.178740 5033 tablet_service.cc:1467] Processing CreateTablet for tablet d01d02061df34778a2587e6e0908ee61 (DEFAULT_TABLE table=test-table [id=744ce97a353a43ab9985a235c2d3521e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:42:49.179481 4723 tablet_service.cc:1467] Processing CreateTablet for tablet d01d02061df34778a2587e6e0908ee61 (DEFAULT_TABLE table=test-table [id=744ce97a353a43ab9985a235c2d3521e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:42:49.180043 5033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01d02061df34778a2587e6e0908ee61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:42:49.180042 4875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01d02061df34778a2587e6e0908ee61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:42:49.180490 4723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01d02061df34778a2587e6e0908ee61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:42:49.186597 5131 tablet_bootstrap.cc:492] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3: Bootstrap starting.
I20260810 03:42:49.188102 5132 tablet_bootstrap.cc:492] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d: Bootstrap starting.
I20260810 03:42:49.188573 5133 tablet_bootstrap.cc:492] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346: Bootstrap starting.
I20260810 03:42:49.189551 5131 tablet_bootstrap.cc:654] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3: Neither blocks nor log segments found. Creating new log.
I20260810 03:42:49.190460 5131 log.cc:824] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3: Log is configured to *not* fsync() on all Append() calls
I20260810 03:42:49.191231 5133 tablet_bootstrap.cc:654] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346: Neither blocks nor log segments found. Creating new log.
I20260810 03:42:49.191747 5132 tablet_bootstrap.cc:654] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d: Neither blocks nor log segments found. Creating new log.
I20260810 03:42:49.192081 5133 log.cc:824] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346: Log is configured to *not* fsync() on all Append() calls
I20260810 03:42:49.192731 5131 tablet_bootstrap.cc:492] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3: No bootstrap required, opened a new log
I20260810 03:42:49.192910 5132 log.cc:824] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d: Log is configured to *not* fsync() on all Append() calls
I20260810 03:42:49.192967 5131 ts_tablet_manager.cc:1397] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 03:42:49.194195 5133 tablet_bootstrap.cc:492] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346: No bootstrap required, opened a new log
I20260810 03:42:49.194380 5133 ts_tablet_manager.cc:1397] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:42:49.194706 5132 tablet_bootstrap.cc:492] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d: No bootstrap required, opened a new log
I20260810 03:42:49.194922 5132 ts_tablet_manager.cc:1397] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 03:42:49.197659 5131 raft_consensus.cc:348] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } } peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } }
I20260810 03:42:49.197940 5131 raft_consensus.cc:374] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:42:49.198057 5131 raft_consensus.cc:729] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b6e6da672e44e2396fc704ec88bcfb3, State: Initialized, Role: FOLLOWER
I20260810 03:42:49.198083 5132 raft_consensus.cc:348] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } } peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } }
I20260810 03:42:49.198392 5132 raft_consensus.cc:374] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:42:49.198271 5133 raft_consensus.cc:348] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } } peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } }
I20260810 03:42:49.198452 5132 raft_consensus.cc:729] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27dd45a0f7d9402aa766cc714ef0c60d, State: Initialized, Role: FOLLOWER
I20260810 03:42:49.198516 5133 raft_consensus.cc:374] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:42:49.198577 5133 raft_consensus.cc:729] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 046f21098e2344e3b444ad86670ab346, State: Initialized, Role: FOLLOWER
I20260810 03:42:49.198529 5131 consensus_queue.cc:260] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } } peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } }
I20260810 03:42:49.198894 5132 consensus_queue.cc:260] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } } peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } }
I20260810 03:42:49.199013 5133 consensus_queue.cc:260] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } } peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } }
I20260810 03:42:49.199213 4799 heartbeater.cc:503] Master 127.21.175.126:36335 was elected leader, sending a full tablet report...
I20260810 03:42:49.199550 5112 heartbeater.cc:503] Master 127.21.175.126:36335 was elected leader, sending a full tablet report...
I20260810 03:42:49.199999 5133 ts_tablet_manager.cc:1428] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 03:42:49.200300 4949 heartbeater.cc:503] Master 127.21.175.126:36335 was elected leader, sending a full tablet report...
I20260810 03:42:49.200701 5132 ts_tablet_manager.cc:1428] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260810 03:42:49.201853 5131 ts_tablet_manager.cc:1428] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260810 03:42:49.272138 5140 raft_consensus.cc:482] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:42:49.272487 5140 raft_consensus.cc:504] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } } peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } }
I20260810 03:42:49.273789 5140 leader_election.cc:288] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27dd45a0f7d9402aa766cc714ef0c60d (127.21.175.67:40255), 046f21098e2344e3b444ad86670ab346 (127.21.175.66:35821)
I20260810 03:42:49.279714 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.274342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:59141 (local address 127.21.175.67:40255)
0810 03:42:49.274511 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:49.274517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:49.274767 (+ 250us) server_negotiation.cc:407] Connection header received
0810 03:42:49.274949 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:49.274952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:49.275006 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:49.275113 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:42:49.275631 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.276442 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.278380 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.279020 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.279141 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.279195 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:49.279496 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 03:42:49.279579 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 03:42:49.279714 5130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.274303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:42991 (local address 127.21.175.66:35821)
0810 03:42:49.274445 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:49.274449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:49.274850 (+ 401us) server_negotiation.cc:407] Connection header received
0810 03:42:49.274940 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:49.274944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:49.275007 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:49.275114 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:42:49.275635 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.276431 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.278379 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.279005 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.279142 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.279195 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:49.279540 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 03:42:49.279609 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:42:49.279798 5145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.274189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:35821 (local address 127.21.175.65:42991)
0810 03:42:49.274741 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:49.274755 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:49.274834 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:49.275139 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:49.275142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:49.275184 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:42:49.275449 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:49.275457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.276583 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.276587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:49.278225 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:49.278239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.279146 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.279149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:49.279334 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.279379 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 03:42:49.279460 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 03:42:49.279595 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":314,"spinlock_wait_cycles":33024,"thread_start_us":202,"threads_started":1}
I20260810 03:42:49.279798 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.274189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:40255 (local address 127.21.175.65:59141)
0810 03:42:49.274654 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:49.274670 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:49.274814 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:49.275139 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:49.275142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:49.275186 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:42:49.275449 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:49.275457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.276583 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.276587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:49.278225 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:49.278238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.279146 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.279149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:49.279361 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.279381 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 03:42:49.279460 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 03:42:49.279595 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":308,"spinlock_wait_cycles":9216,"thread_start_us":140,"threads_started":1}
I20260810 03:42:49.280562 5056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01d02061df34778a2587e6e0908ee61" candidate_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" is_pre_election: true
I20260810 03:42:49.280567 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01d02061df34778a2587e6e0908ee61" candidate_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "046f21098e2344e3b444ad86670ab346" is_pre_election: true
I20260810 03:42:49.280931 5056 raft_consensus.cc:2454] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b6e6da672e44e2396fc704ec88bcfb3 in term 0.
I20260810 03:42:49.280931 4898 raft_consensus.cc:2454] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b6e6da672e44e2396fc704ec88bcfb3 in term 0.
I20260810 03:42:49.281569 4667 leader_election.cc:302] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 046f21098e2344e3b444ad86670ab346, 27dd45a0f7d9402aa766cc714ef0c60d, 4b6e6da672e44e2396fc704ec88bcfb3; no voters:
I20260810 03:42:49.281889 5140 raft_consensus.cc:2781] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:42:49.282009 5140 raft_consensus.cc:482] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:42:49.282094 5140 raft_consensus.cc:3037] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:42:49.283245 5140 raft_consensus.cc:504] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } } peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } }
I20260810 03:42:49.283644 5140 leader_election.cc:288] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 1 election: Requested vote from peers 27dd45a0f7d9402aa766cc714ef0c60d (127.21.175.67:40255), 046f21098e2344e3b444ad86670ab346 (127.21.175.66:35821)
I20260810 03:42:49.284165 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01d02061df34778a2587e6e0908ee61" candidate_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "046f21098e2344e3b444ad86670ab346"
I20260810 03:42:49.284154 5056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01d02061df34778a2587e6e0908ee61" candidate_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27dd45a0f7d9402aa766cc714ef0c60d"
I20260810 03:42:49.284298 5056 raft_consensus.cc:3037] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:42:49.284298 4898 raft_consensus.cc:3037] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:42:49.285742 5056 raft_consensus.cc:2454] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b6e6da672e44e2396fc704ec88bcfb3 in term 1.
I20260810 03:42:49.285742 4898 raft_consensus.cc:2454] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b6e6da672e44e2396fc704ec88bcfb3 in term 1.
I20260810 03:42:49.286126 4667 leader_election.cc:302] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 046f21098e2344e3b444ad86670ab346, 27dd45a0f7d9402aa766cc714ef0c60d, 4b6e6da672e44e2396fc704ec88bcfb3; no voters:
I20260810 03:42:49.286360 5140 raft_consensus.cc:2781] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:42:49.286640 5140 raft_consensus.cc:686] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 LEADER]: Becoming Leader. State: Replica: 4b6e6da672e44e2396fc704ec88bcfb3, State: Running, Role: LEADER
I20260810 03:42:49.287060 5140 consensus_queue.cc:237] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } } peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } }
I20260810 03:42:49.290516 4576 catalog_manager.cc:5495] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b6e6da672e44e2396fc704ec88bcfb3 (127.21.175.65). New cstate: current_term: 1 leader_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } health_report { overall_health: UNKNOWN } } }
I20260810 03:42:49.327505 5128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.323715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34986 (local address 127.21.175.65:35749)
0810 03:42:49.323859 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:49.323862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:49.323877 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:42:49.324022 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:49.324025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:49.324082 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:49.324181 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:42:49.324442 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.324985 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.325990 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.326214 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.326296 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.326608 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:42:49.327042 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:42:49.327137 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:49.327273 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 03:42:49.327328 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 03:42:49.333847 4898 raft_consensus.cc:1261] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346 [term 1 FOLLOWER]: Refusing update from remote peer 4b6e6da672e44e2396fc704ec88bcfb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:42:49.333847 5056 raft_consensus.cc:1261] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d [term 1 FOLLOWER]: Refusing update from remote peer 4b6e6da672e44e2396fc704ec88bcfb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:42:49.334839 5140 consensus_queue.cc:1035] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:42:49.335071 5146 consensus_queue.cc:1035] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:42:49.344009 5150 mvcc.cc:205] Tried to move back new op lower bound from 7316821480783192064 to 7316821480602894336. Current Snapshot: MvccSnapshot[applied={T|T < 7316821480783192064}]
I20260810 03:42:49.345007 5151 mvcc.cc:205] Tried to move back new op lower bound from 7316821480783192064 to 7316821480602894336. Current Snapshot: MvccSnapshot[applied={T|T < 7316821480783192064}]
I20260810 03:42:49.354247 4570 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:42:49.354416 4570 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:42:49.357066 4570 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734: Sending DeleteTablet for 3 replicas of tablet d01d02061df34778a2587e6e0908ee61
I20260810 03:42:49.357831 4723 tablet_service.cc:1514] Processing DeleteTablet for tablet d01d02061df34778a2587e6e0908ee61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:42:49 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:34980
I20260810 03:42:49.357894 5033 tablet_service.cc:1514] Processing DeleteTablet for tablet d01d02061df34778a2587e6e0908ee61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:42:49 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:33268
I20260810 03:42:49.358844 4875 tablet_service.cc:1514] Processing DeleteTablet for tablet d01d02061df34778a2587e6e0908ee61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:42:49 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:49962
I20260810 03:42:49.359036 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 4647
I20260810 03:42:49.361382 5164 tablet_replica.cc:330] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d: stopping tablet replica
I20260810 03:42:49.361778 5164 raft_consensus.cc:2229] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:42:49.362169 5164 raft_consensus.cc:2258] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:42:49.362757 5165 tablet_replica.cc:330] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346: stopping tablet replica
I20260810 03:42:49.363135 5165 raft_consensus.cc:2229] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:42:49.363457 5165 raft_consensus.cc:2258] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:42:49.364812 5165 ts_tablet_manager.cc:1905] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:42:49.365195 5164 ts_tablet_manager.cc:1905] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:42:49.367220 5165 ts_tablet_manager.cc:1918] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:42:49.367344 5165 log.cc:1196] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/d01d02061df34778a2587e6e0908ee61
I20260810 03:42:49.367671 5165 ts_tablet_manager.cc:1939] T d01d02061df34778a2587e6e0908ee61 P 046f21098e2344e3b444ad86670ab346: Deleting consensus metadata
I20260810 03:42:49.369045 5164 ts_tablet_manager.cc:1918] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:42:49.369163 5164 log.cc:1196] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/d01d02061df34778a2587e6e0908ee61
I20260810 03:42:49.369182 4561 catalog_manager.cc:4841] TS 046f21098e2344e3b444ad86670ab346 (127.21.175.66:35821): tablet d01d02061df34778a2587e6e0908ee61 (table test-table [id=744ce97a353a43ab9985a235c2d3521e]) successfully deleted
I20260810 03:42:49.369711 5164 ts_tablet_manager.cc:1939] T d01d02061df34778a2587e6e0908ee61 P 27dd45a0f7d9402aa766cc714ef0c60d: Deleting consensus metadata
W20260810 03:42:49.371352 4561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.175.65:35749 (error 108)
I20260810 03:42:49.371382 4563 catalog_manager.cc:4841] TS 27dd45a0f7d9402aa766cc714ef0c60d (127.21.175.67:40255): tablet d01d02061df34778a2587e6e0908ee61 (table test-table [id=744ce97a353a43ab9985a235c2d3521e]) successfully deleted
I20260810 03:42:49.371613 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 4804
I20260810 03:42:49.375159 5129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.374858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:35749 (local address 127.0.0.1:34988)
0810 03:42:49.375000 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:49.375078 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 03:42:49.375388 4561 catalog_manager.cc:4584] TS 4b6e6da672e44e2396fc704ec88bcfb3 (127.21.175.65:35749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d01d02061df34778a2587e6e0908ee61: Network error: Client connection negotiation failed: client connection to 127.21.175.65:35749: connect: Connection refused (error 111)
I20260810 03:42:49.383319 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 4956
I20260810 03:42:49.395612 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 4541
I20260810 03:42:49.404297 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:36335
--webserver_interface=127.21.175.126
--webserver_port=38097
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:36335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:42:49.526423 5167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:49.526711 5167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:49.526753 5167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:49.531693 5167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:42:49.531795 5167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:49.531831 5167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:49.531874 5167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:42:49.531910 5167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:42:49.540125 5167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:36335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:36335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=38097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:49.541450 5167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:49.542593 5167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:49.557329 5181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:49.557332 5179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:49.557435 5167 server_base.cc:1034] running on GCE node
W20260810 03:42:49.557329 5178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:49.558405 5167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:49.559765 5167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:49.560971 5167 hybrid_clock.cc:648] HybridClock initialized: now 1786333369560945 us; error 40 us; skew 500 ppm
Aug 10 03:42:49 dist-test-slave-ss4n krb5kdc[4526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:49.564325 5167 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:42:49.565359 5167 webserver.cc:460] Webserver started at http://127.21.175.126:38097/ using document root <none> and password file <none>
I20260810 03:42:49.566031 5167 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:49.566118 5167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:49.570390 5167 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:42:49.572885 5188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:49.574318 5167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:42:49.574470 5167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "4de9004f7e474d2db46d3fa2e8170734"
format_stamp: "Formatted at 2026-08-10 03:42:48 on dist-test-slave-ss4n"
server_key: "bf9f7c3463406d3783f9cb5b8289f152"
server_key_iv: "b07a16796e47e4ed767945e9bd3f16e6"
server_key_version: "encryptionkey@0"
I20260810 03:42:49.575158 5167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:49.588359 5167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:49.592437 5167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:49.592693 5167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:49.600792 5167 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:36335
I20260810 03:42:49.601114 5246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:36335 every 8 connection(s)
I20260810 03:42:49.602241 5167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:42:49.606858 5247 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:42:49.608558 5247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734: Bootstrap starting.
I20260810 03:42:49.612355 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 5167
I20260810 03:42:49.612841 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:35749
--local_ip_for_outbound_sockets=127.21.175.65
--tserver_master_addrs=127.21.175.126:36335
--webserver_port=36477
--webserver_interface=127.21.175.65
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:42:49.626376 5247 log.cc:824] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734: Log is configured to *not* fsync() on all Append() calls
I20260810 03:42:49.635429 5247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:42:49.635834 5247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734: Bootstrap complete.
I20260810 03:42:49.640093 5247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } }
I20260810 03:42:49.640516 5247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de9004f7e474d2db46d3fa2e8170734, State: Initialized, Role: FOLLOWER
I20260810 03:42:49.641059 5247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } }
I20260810 03:42:49.641207 5247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:42:49.641305 5247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:42:49.641428 5247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:42:49.643782 5247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } }
I20260810 03:42:49.644124 5247 leader_election.cc:302] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4de9004f7e474d2db46d3fa2e8170734; no voters:
I20260810 03:42:49.644481 5247 leader_election.cc:288] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 03:42:49.644577 5252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:42:49.644779 5252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 2 LEADER]: Becoming Leader. State: Replica: 4de9004f7e474d2db46d3fa2e8170734, State: Running, Role: LEADER
I20260810 03:42:49.645088 5252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } }
I20260810 03:42:49.645870 5247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:42:49.646150 5252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4de9004f7e474d2db46d3fa2e8170734. Latest consensus state: current_term: 2 leader_uuid: "4de9004f7e474d2db46d3fa2e8170734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } } }
I20260810 03:42:49.646274 5252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [sys.catalog]: This master's current role is: LEADER
I20260810 03:42:49.646656 5256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:42:49.646823 5253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4de9004f7e474d2db46d3fa2e8170734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } } }
I20260810 03:42:49.647027 5253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [sys.catalog]: This master's current role is: LEADER
I20260810 03:42:49.651612 5256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:42:49.652268 5256 catalog_manager.cc:1259] Loaded cluster ID: a329cb1f97c446ba8b0a861c6e2e98f6
I20260810 03:42:49.652333 5256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:42:49.653781 5256 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 03:42:49.659255 5256 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 03:42:49.665163 5256 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:42:49.666126 5256 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734: Loaded TSK: 0
I20260810 03:42:49.667744 5256 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:42:49.747255 5249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:49.747591 5249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:49.747676 5249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:49.752406 5249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:49.752478 5249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:49.752597 5249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:42:49.758682 5249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:35749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=36477
--tserver_master_addrs=127.21.175.126:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:49.759801 5249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:49.760897 5249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:49.771857 5283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:49.771883 5280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:49.771904 5281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:49.772320 5249 server_base.cc:1034] running on GCE node
I20260810 03:42:49.772780 5249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:49.773458 5249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:49.774709 5249 hybrid_clock.cc:648] HybridClock initialized: now 1786333369774692 us; error 38 us; skew 500 ppm
Aug 10 03:42:49 dist-test-slave-ss4n krb5kdc[4526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:49.777791 5249 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:42:49.778836 5249 webserver.cc:460] Webserver started at http://127.21.175.65:36477/ using document root <none> and password file <none>
I20260810 03:42:49.779424 5249 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:49.779502 5249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:49.783715 5249 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 03:42:49.786141 5290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:49.787706 5249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:42:49.787864 5249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "4b6e6da672e44e2396fc704ec88bcfb3"
format_stamp: "Formatted at 2026-08-10 03:42:48 on dist-test-slave-ss4n"
server_key: "df02fffb25683b22cd43fb159702ac60"
server_key_iv: "9b00ce6b16e604266f0d69bb2f648614"
server_key_version: "encryptionkey@0"
I20260810 03:42:49.788450 5249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:49.806869 5249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:49.810431 5249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:49.810659 5249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:49.811300 5249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:42:49.812425 5299 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 03:42:49.817014 5249 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 03:42:49.817134 5249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 03:42:49.817235 5249 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 03:42:49.819195 5249 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 03:42:49.819280 5249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260810 03:42:49.819653 5299 tablet_bootstrap.cc:492] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3: Bootstrap starting.
I20260810 03:42:49.831089 5249 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:35749
I20260810 03:42:49.831185 5420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:35749 every 8 connection(s)
I20260810 03:42:49.832191 5249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:42:49.840538 5299 log.cc:824] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3: Log is configured to *not* fsync() on all Append() calls
I20260810 03:42:49.841581 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 5249
I20260810 03:42:49.842253 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:35821
--local_ip_for_outbound_sockets=127.21.175.66
--tserver_master_addrs=127.21.175.126:36335
--webserver_port=41145
--webserver_interface=127.21.175.66
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:42:49 dist-test-slave-ss4n krb5kdc[4526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:42:49.852185 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.834541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:56893)
0810 03:42:49.835230 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:49.835283 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:49.836192 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:49.837761 (+ 1569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:49.837769 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:49.838162 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:49.838673 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:49.838689 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.840551 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.840554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:49.841861 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:49.841873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.843153 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.843157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:49.843379 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.844098 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:49.844126 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:49.846351 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:49.848918 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:49.848926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:49.848940 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:49.849218 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:49.849655 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:49.849659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:49.849661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:49.849806 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:49.850410 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:49.850416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:49.850672 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 03:42:49.850853 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 03:42:49.851240 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":134,"threads_started":1}
I20260810 03:42:49.852171 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.834622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:56893 (local address 127.21.175.126:36335)
0810 03:42:49.835054 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:49.835083 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:49.835430 (+ 347us) server_negotiation.cc:407] Connection header received
0810 03:42:49.836706 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:49.836726 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:49.837188 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:49.837611 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:49.838863 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.840362 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.842021 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.842733 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.842926 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.846629 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:49.846635 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:49.846649 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:49.846681 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:49.848781 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:49.849353 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:49.849364 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:49.849368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:49.849548 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:49.849912 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:49.849915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:49.849917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:49.850289 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:49.850441 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:49.850987 (+ 546us) server_negotiation.cc:299] Negotiation successful
0810 03:42:49.851251 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":153,"threads_started":1}
I20260810 03:42:49.853435 5299 tablet_bootstrap.cc:492] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:42:49.854001 5299 tablet_bootstrap.cc:492] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3: Bootstrap complete.
I20260810 03:42:49.854836 5423 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36335
I20260810 03:42:49.855079 5423 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:49.855552 5299 ts_tablet_manager.cc:1397] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3: Time spent bootstrapping tablet: real 0.036s user 0.031s sys 0.000s
I20260810 03:42:49.855589 5423 heartbeater.cc:511] Master 127.21.175.126:36335 requested a full tablet report, sending...
I20260810 03:42:49.858480 5208 ts_manager.cc:194] Registered new tserver with Master: 4b6e6da672e44e2396fc704ec88bcfb3 (127.21.175.65:35749)
I20260810 03:42:49.858844 5299 raft_consensus.cc:348] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } } peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } }
I20260810 03:42:49.859400 5299 raft_consensus.cc:729] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b6e6da672e44e2396fc704ec88bcfb3, State: Initialized, Role: FOLLOWER
I20260810 03:42:49.859895 5299 consensus_queue.cc:260] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } } peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } }
I20260810 03:42:49.860941 5299 ts_tablet_manager.cc:1428] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260810 03:42:49.862718 5208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:56893
I20260810 03:42:49.865439 5423 heartbeater.cc:503] Master 127.21.175.126:36335 was elected leader, sending a full tablet report...
Aug 10 03:42:49 dist-test-slave-ss4n krb5kdc[4526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for kudu/127.21.175.65@KRBTEST.COM
I20260810 03:42:49.874194 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.861762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:35749 (local address 127.0.0.1:35002)
0810 03:42:49.862204 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:49.862253 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:49.862379 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:49.864131 (+ 1752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:49.864137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:49.864589 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:49.864723 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:49.864732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.866052 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.866056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:49.867021 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:49.867033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.867841 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.867844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:49.867927 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.868643 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:49.868663 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:49.870204 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:49.872397 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:49.872405 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:49.872410 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:49.872665 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:49.872991 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:49.872999 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:49.873001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:49.873079 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:49.873647 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:49.873652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:49.873759 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 03:42:49.873877 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 03:42:49.874062 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":103,"threads_started":1}
I20260810 03:42:49.874325 5432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:49.861821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35002 (local address 127.21.175.65:35749)
0810 03:42:49.863634 (+ 1813us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:49.863644 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:49.863664 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:42:49.863722 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:49.863726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:49.863948 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:49.864092 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:49.864876 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.865909 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.867155 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:49.867713 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:49.867830 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:49.870347 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:49.870358 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:49.870372 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:49.870399 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:49.872265 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:49.872784 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:49.872790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:49.872792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:49.872888 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:49.873167 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:49.873170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:49.873172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:49.873466 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:49.873661 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:49.873991 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 03:42:49.874163 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1713,"thread_start_us":129,"threads_started":1}
I20260810 03:42:49.875417 5350 tablet_service.cc:1514] Processing DeleteTablet for tablet d01d02061df34778a2587e6e0908ee61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:42:49 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:35002
I20260810 03:42:49.876403 5433 tablet_replica.cc:330] stopping tablet replica
I20260810 03:42:49.876704 5433 raft_consensus.cc:2229] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:42:49.877024 5433 raft_consensus.cc:2258] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:42:49.878240 5433 ts_tablet_manager.cc:1905] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:42:49.882025 5433 ts_tablet_manager.cc:1918] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:42:49.882164 5433 log.cc:1196] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal/wals/d01d02061df34778a2587e6e0908ee61
I20260810 03:42:49.882467 5433 ts_tablet_manager.cc:1939] T d01d02061df34778a2587e6e0908ee61 P 4b6e6da672e44e2396fc704ec88bcfb3: Deleting consensus metadata
I20260810 03:42:49.884187 5190 catalog_manager.cc:4841] TS 4b6e6da672e44e2396fc704ec88bcfb3 (127.21.175.65:35749): tablet d01d02061df34778a2587e6e0908ee61 (table test-table [id=744ce97a353a43ab9985a235c2d3521e]) successfully deleted
W20260810 03:42:49.971735 5429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:49.972015 5429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:49.972062 5429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:49.976255 5429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:49.976323 5429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:49.976418 5429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:42:49.982342 5429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:35821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=41145
--tserver_master_addrs=127.21.175.126:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:49.983510 5429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:49.984622 5429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:49.995474 5439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:49.995476 5438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:49.995498 5441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:49.995805 5429 server_base.cc:1034] running on GCE node
I20260810 03:42:49.996276 5429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:49.997004 5429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:49.998204 5429 hybrid_clock.cc:648] HybridClock initialized: now 1786333369998188 us; error 27 us; skew 500 ppm
Aug 10 03:42:49 dist-test-slave-ss4n krb5kdc[4526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:50.001170 5429 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:42:50.002133 5429 webserver.cc:460] Webserver started at http://127.21.175.66:41145/ using document root <none> and password file <none>
I20260810 03:42:50.002683 5429 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:50.002758 5429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:50.006567 5429 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 03:42:50.008769 5450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:50.010154 5429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:42:50.010257 5429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "046f21098e2344e3b444ad86670ab346"
format_stamp: "Formatted at 2026-08-10 03:42:48 on dist-test-slave-ss4n"
server_key: "9d38b2b572fd77037a811de2822de28a"
server_key_iv: "e02ecf8d3ce8b28d82e48a72950e7c15"
server_key_version: "encryptionkey@0"
I20260810 03:42:50.010823 5429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:50.030975 5429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:50.034756 5429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:50.034991 5429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:50.035632 5429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:42:50.036536 5429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:42:50.036630 5429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:50.036708 5429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:42:50.036773 5429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:50.046648 5429 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:35821
I20260810 03:42:50.046722 5583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:35821 every 8 connection(s)
I20260810 03:42:50.047876 5429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:42:50.052090 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 5429
I20260810 03:42:50.052631 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:40255
--local_ip_for_outbound_sockets=127.21.175.67
--tserver_master_addrs=127.21.175.126:36335
--webserver_port=36155
--webserver_interface=127.21.175.67
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:42:50 dist-test-slave-ss4n krb5kdc[4526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:42:50.064604 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:50.050029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:44041 (local address 127.21.175.126:36335)
0810 03:42:50.050189 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:50.050193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:50.051071 (+ 878us) server_negotiation.cc:407] Connection header received
0810 03:42:50.053099 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:50.053102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:50.053160 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:50.053227 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:50.055098 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.055702 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.057030 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.057310 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.057454 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:50.060367 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:50.060372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:50.060375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:50.060413 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:50.062185 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:50.062807 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:50.062816 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:50.062818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:50.062905 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:50.063328 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:50.063334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:50.063338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:50.063562 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:50.063683 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:50.064257 (+ 574us) server_negotiation.cc:299] Negotiation successful
0810 03:42:50.064418 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:42:50.065641 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:50.050325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:44041)
0810 03:42:50.050844 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:50.050895 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:50.052810 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:50.053470 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:50.053505 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:50.054130 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:50.054915 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:50.054935 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.055857 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.055866 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:50.056873 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:50.056884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.057537 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.057542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:50.057739 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:50.058447 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:50.058467 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:50.060150 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:50.062336 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:50.062343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:50.062364 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:50.062675 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:50.063039 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:50.063043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:50.063045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:50.063217 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:50.063696 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:50.063706 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:50.063995 (+ 289us) client_negotiation.cc:769] Sending connection context
0810 03:42:50.064211 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 03:42:50.064600 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":148,"threads_started":1}
I20260810 03:42:50.067076 5584 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36335
I20260810 03:42:50.067366 5584 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:50.067929 5584 heartbeater.cc:511] Master 127.21.175.126:36335 requested a full tablet report, sending...
I20260810 03:42:50.069183 5208 ts_manager.cc:194] Registered new tserver with Master: 046f21098e2344e3b444ad86670ab346 (127.21.175.66:35821)
I20260810 03:42:50.070046 5208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:44041
W20260810 03:42:50.172142 5587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:50.172410 5587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:50.172452 5587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:50.176460 5587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:50.176527 5587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:50.176586 5587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:42:50.182376 5587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:40255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=36155
--tserver_master_addrs=127.21.175.126:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:50.183430 5587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:50.184461 5587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:50.195173 5594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:50.195154 5595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:50.195138 5587 server_base.cc:1034] running on GCE node
W20260810 03:42:50.196262 5597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:50.196662 5587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:50.197273 5587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:50.198455 5587 hybrid_clock.cc:648] HybridClock initialized: now 1786333370198434 us; error 38 us; skew 500 ppm
Aug 10 03:42:50 dist-test-slave-ss4n krb5kdc[4526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333370, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:50.201774 5587 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:42:50.202858 5587 webserver.cc:460] Webserver started at http://127.21.175.67:36155/ using document root <none> and password file <none>
I20260810 03:42:50.203497 5587 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:50.203591 5587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:50.208020 5587 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:42:50.210500 5606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:50.211947 5587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:42:50.212124 5587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "27dd45a0f7d9402aa766cc714ef0c60d"
format_stamp: "Formatted at 2026-08-10 03:42:49 on dist-test-slave-ss4n"
server_key: "302fb5c905eeb5962c7ce28c33107ebf"
server_key_iv: "f018eb3c40744f81bb5178512541a3a2"
server_key_version: "encryptionkey@0"
I20260810 03:42:50.212785 5587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:50.237916 5587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:50.241245 5587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:50.241509 5587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:50.242177 5587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:42:50.243112 5587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:42:50.243196 5587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:50.243273 5587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:42:50.243314 5587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:50.253305 5587 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:40255
I20260810 03:42:50.253329 5738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:40255 every 8 connection(s)
I20260810 03:42:50.254452 5587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:42:50.261127 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 5587
Aug 10 03:42:50 dist-test-slave-ss4n krb5kdc[4526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333370, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:42:50.270269 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:50.256566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:51089 (local address 127.21.175.126:36335)
0810 03:42:50.256768 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:50.256774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:50.257694 (+ 920us) server_negotiation.cc:407] Connection header received
0810 03:42:50.259072 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:50.259076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:50.259129 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:50.259217 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:50.260524 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.261446 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.262940 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.263219 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.263340 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:50.266160 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:50.266166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:50.266173 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:50.266224 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:50.267700 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:50.268338 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:50.268345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:50.268349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:50.268478 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:50.268970 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:50.268974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:50.268976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:50.269294 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:50.269414 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:50.270014 (+ 600us) server_negotiation.cc:299] Negotiation successful
0810 03:42:50.270135 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"spinlock_wait_cycles":2944}
I20260810 03:42:50.271488 5741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:50.256942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:51089)
0810 03:42:50.257427 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:50.257578 (+ 151us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:50.258853 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:50.259424 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:50.259431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:50.259828 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:50.260309 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:50.260323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.261628 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.261632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:50.262803 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:50.262814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.263379 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.263383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:50.263571 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:50.264222 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:50.264239 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:50.265983 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:50.267843 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:50.267849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:50.267866 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:50.268204 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:50.268653 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:50.268655 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:50.268657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:50.268839 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:50.269439 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:50.269445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:50.269763 (+ 318us) client_negotiation.cc:769] Sending connection context
0810 03:42:50.269931 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 03:42:50.270326 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":130,"threads_started":1}
I20260810 03:42:50.271814 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:50.262007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.21.175.126:36335)
0810 03:42:50.262323 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:50.262336 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:50.262484 (+ 148us) server_negotiation.cc:407] Connection header received
0810 03:42:50.262646 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:50.262650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:50.262707 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:50.262816 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:50.263614 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.264368 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.265753 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.266013 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.266137 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:50.267884 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:50.267888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:50.267890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:50.267913 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:50.269863 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:50.270398 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:50.270402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:50.270404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:50.270491 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:50.270798 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:50.270801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:50.270803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:50.271002 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:50.271229 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:50.271388 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 03:42:50.271512 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"spinlock_wait_cycles":1408,"thread_start_us":130,"threads_started":1}
I20260810 03:42:50.272796 5739 heartbeater.cc:347] Connected to a master server at 127.21.175.126:36335
I20260810 03:42:50.273125 5739 heartbeater.cc:464] Registering TS with master...
I20260810 03:42:50.273792 5739 heartbeater.cc:511] Master 127.21.175.126:36335 requested a full tablet report, sending...
I20260810 03:42:50.275094 5207 ts_manager.cc:194] Registered new tserver with Master: 27dd45a0f7d9402aa766cc714ef0c60d (127.21.175.67:40255)
I20260810 03:42:50.275772 5207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:51089
I20260810 03:42:50.286552 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:42:50.291306 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:50.287337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.21.175.126:36335)
0810 03:42:50.287497 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:50.287501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:50.287752 (+ 251us) server_negotiation.cc:407] Connection header received
0810 03:42:50.287865 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:50.287867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:50.287995 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:50.288102 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:42:50.288332 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.288931 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.289840 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.290096 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.290204 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:50.290473 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:42:50.290960 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:42:50.291067 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:50.291131 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 03:42:50.291189 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:42:50.292150 5207 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:42:50.295894 5207 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:42:50.304674 5350 tablet_service.cc:1467] Processing CreateTablet for tablet 29ddbc910cd24369ad701a35084487a7 (DEFAULT_TABLE table=test-table [id=fa9bd0299334437087ec6ca3027432d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:42:50.305445 5350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29ddbc910cd24369ad701a35084487a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:42:50.308689 5750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:50.303432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:40255 (local address 127.0.0.1:33274)
0810 03:42:50.303948 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:50.303966 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:50.304057 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:50.304813 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:50.304816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:50.304834 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:42:50.304924 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:50.304930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.306429 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.306433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:50.307428 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:50.307436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.308234 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.308238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:50.308332 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:50.308379 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 03:42:50.308454 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 03:42:50.308533 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":155,"threads_started":1}
I20260810 03:42:50.308838 5299 tablet_bootstrap.cc:492] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3: Bootstrap starting.
I20260810 03:42:50.309118 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:50.303251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:35821 (local address 127.0.0.1:49972)
0810 03:42:50.303537 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:50.303559 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:50.303660 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:50.304422 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:50.304425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:50.304444 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:42:50.304571 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:50.304582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.305924 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.305928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:50.307009 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:50.307020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.308288 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.308293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:50.308410 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:50.308449 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 03:42:50.308518 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 03:42:50.308606 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260810 03:42:50.309547 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:50.303955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.21.175.67:40255)
0810 03:42:50.304348 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:50.304361 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:50.304386 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:42:50.304469 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:50.304475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:50.304661 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:50.304834 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:42:50.305023 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.306273 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.307563 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.308105 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.308232 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:50.308389 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:50.308484 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 03:42:50.308583 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":127,"threads_started":1}
I20260810 03:42:50.309684 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:50.303436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49972 (local address 127.21.175.66:35821)
0810 03:42:50.303848 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:50.303862 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:50.303890 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:42:50.303977 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:50.303985 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:50.304250 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:50.304446 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:42:50.304706 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.305791 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.307203 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.308119 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.308771 (+ 652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:50.308940 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:50.309000 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 03:42:50.309102 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":167,"threads_started":1}
I20260810 03:42:50.309873 5299 tablet_bootstrap.cc:654] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3: Neither blocks nor log segments found. Creating new log.
I20260810 03:42:50.311321 5299 tablet_bootstrap.cc:492] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3: No bootstrap required, opened a new log
I20260810 03:42:50.311422 5299 ts_tablet_manager.cc:1397] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:42:50.311882 5299 raft_consensus.cc:348] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } } peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } }
I20260810 03:42:50.312011 5299 raft_consensus.cc:374] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:42:50.312070 5299 raft_consensus.cc:729] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b6e6da672e44e2396fc704ec88bcfb3, State: Initialized, Role: FOLLOWER
I20260810 03:42:50.312230 5299 consensus_queue.cc:260] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } } peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } }
I20260810 03:42:50.312498 5299 ts_tablet_manager.cc:1428] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:42:50.312942 5658 tablet_service.cc:1467] Processing CreateTablet for tablet 29ddbc910cd24369ad701a35084487a7 (DEFAULT_TABLE table=test-table [id=fa9bd0299334437087ec6ca3027432d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:42:50.312948 5509 tablet_service.cc:1467] Processing CreateTablet for tablet 29ddbc910cd24369ad701a35084487a7 (DEFAULT_TABLE table=test-table [id=fa9bd0299334437087ec6ca3027432d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:42:50.314044 5658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29ddbc910cd24369ad701a35084487a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:42:50.314083 5509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29ddbc910cd24369ad701a35084487a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:42:50.321548 5754 tablet_bootstrap.cc:492] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d: Bootstrap starting.
I20260810 03:42:50.321619 5755 tablet_bootstrap.cc:492] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346: Bootstrap starting.
I20260810 03:42:50.324107 5755 tablet_bootstrap.cc:654] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346: Neither blocks nor log segments found. Creating new log.
I20260810 03:42:50.324231 5754 tablet_bootstrap.cc:654] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d: Neither blocks nor log segments found. Creating new log.
I20260810 03:42:50.325089 5755 log.cc:824] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346: Log is configured to *not* fsync() on all Append() calls
I20260810 03:42:50.325155 5754 log.cc:824] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d: Log is configured to *not* fsync() on all Append() calls
I20260810 03:42:50.327077 5755 tablet_bootstrap.cc:492] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346: No bootstrap required, opened a new log
I20260810 03:42:50.327097 5754 tablet_bootstrap.cc:492] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d: No bootstrap required, opened a new log
I20260810 03:42:50.327304 5755 ts_tablet_manager.cc:1397] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260810 03:42:50.327316 5754 ts_tablet_manager.cc:1397] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260810 03:42:50.330437 5755 raft_consensus.cc:348] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } } peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } }
I20260810 03:42:50.330439 5754 raft_consensus.cc:348] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } } peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } }
I20260810 03:42:50.330741 5755 raft_consensus.cc:374] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:42:50.330741 5754 raft_consensus.cc:374] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:42:50.330801 5754 raft_consensus.cc:729] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27dd45a0f7d9402aa766cc714ef0c60d, State: Initialized, Role: FOLLOWER
I20260810 03:42:50.330801 5755 raft_consensus.cc:729] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 046f21098e2344e3b444ad86670ab346, State: Initialized, Role: FOLLOWER
I20260810 03:42:50.331333 5755 consensus_queue.cc:260] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } } peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } }
I20260810 03:42:50.331347 5754 consensus_queue.cc:260] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } } peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } }
I20260810 03:42:50.332031 5584 heartbeater.cc:503] Master 127.21.175.126:36335 was elected leader, sending a full tablet report...
I20260810 03:42:50.332058 5739 heartbeater.cc:503] Master 127.21.175.126:36335 was elected leader, sending a full tablet report...
I20260810 03:42:50.332412 5755 ts_tablet_manager.cc:1428] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260810 03:42:50.332428 5754 ts_tablet_manager.cc:1428] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260810 03:42:50.333485 5424 tablet.cc:2369] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:42:50.465991 5758 raft_consensus.cc:482] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:42:50.466209 5758 raft_consensus.cc:504] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } } peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } }
I20260810 03:42:50.467437 5758 leader_election.cc:288] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27dd45a0f7d9402aa766cc714ef0c60d (127.21.175.67:40255), 4b6e6da672e44e2396fc704ec88bcfb3 (127.21.175.65:35749)
I20260810 03:42:50.472818 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:50.467779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:40255 (local address 127.21.175.66:36021)
0810 03:42:50.467900 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:50.467918 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:50.468047 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:50.468456 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:50.468458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:50.468471 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:42:50.468567 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:50.468572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.469671 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.469674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:50.471284 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:50.471297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.472221 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.472228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:50.472362 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:50.472418 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 03:42:50.472524 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 03:42:50.472628 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":59}
I20260810 03:42:50.472849 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:50.467877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:36021 (local address 127.21.175.67:40255)
0810 03:42:50.468016 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:50.468021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:50.468035 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:42:50.468189 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:50.468192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:50.468243 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:50.468319 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:42:50.468725 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.469546 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.471443 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.472048 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.472190 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:50.472256 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:50.472628 (+ 372us) server_negotiation.cc:299] Negotiation successful
0810 03:42:50.472721 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:42:50.472887 5764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:50.468135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:51335 (local address 127.21.175.65:35749)
0810 03:42:50.468421 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:50.468424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:50.468438 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:42:50.468487 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:50.468489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:50.468544 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:50.468641 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:42:50.468921 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.469724 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.471443 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.472152 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.472258 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:50.472408 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:50.472671 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 03:42:50.472753 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":121,"threads_started":1}
I20260810 03:42:50.472831 5763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:50.467856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:35749 (local address 127.21.175.66:51335)
0810 03:42:50.468236 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:50.468249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:50.468326 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:50.468676 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:50.468680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:50.468701 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:42:50.468804 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:50.468808 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.469842 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.469845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:50.471284 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:50.471297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.472282 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.472285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:50.472504 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:50.472540 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 03:42:50.472615 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 03:42:50.472673 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":83,"thread_start_us":143,"threads_started":1}
I20260810 03:42:50.473474 5683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29ddbc910cd24369ad701a35084487a7" candidate_uuid: "046f21098e2344e3b444ad86670ab346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" is_pre_election: true
I20260810 03:42:50.473476 5373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29ddbc910cd24369ad701a35084487a7" candidate_uuid: "046f21098e2344e3b444ad86670ab346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" is_pre_election: true
I20260810 03:42:50.473841 5683 raft_consensus.cc:2454] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 046f21098e2344e3b444ad86670ab346 in term 0.
I20260810 03:42:50.473841 5373 raft_consensus.cc:2454] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 046f21098e2344e3b444ad86670ab346 in term 0.
I20260810 03:42:50.474459 5453 leader_election.cc:302] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 046f21098e2344e3b444ad86670ab346, 4b6e6da672e44e2396fc704ec88bcfb3; no voters:
I20260810 03:42:50.474766 5758 raft_consensus.cc:2781] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:42:50.474860 5758 raft_consensus.cc:482] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:42:50.474893 5758 raft_consensus.cc:3037] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:42:50.476012 5758 raft_consensus.cc:504] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } } peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } }
I20260810 03:42:50.476402 5758 leader_election.cc:288] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [CANDIDATE]: Term 1 election: Requested vote from peers 27dd45a0f7d9402aa766cc714ef0c60d (127.21.175.67:40255), 4b6e6da672e44e2396fc704ec88bcfb3 (127.21.175.65:35749)
I20260810 03:42:50.476925 5373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29ddbc910cd24369ad701a35084487a7" candidate_uuid: "046f21098e2344e3b444ad86670ab346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b6e6da672e44e2396fc704ec88bcfb3"
I20260810 03:42:50.476925 5683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29ddbc910cd24369ad701a35084487a7" candidate_uuid: "046f21098e2344e3b444ad86670ab346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27dd45a0f7d9402aa766cc714ef0c60d"
I20260810 03:42:50.477077 5683 raft_consensus.cc:3037] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:42:50.477077 5373 raft_consensus.cc:3037] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:42:50.478451 5683 raft_consensus.cc:2454] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 046f21098e2344e3b444ad86670ab346 in term 1.
I20260810 03:42:50.478451 5373 raft_consensus.cc:2454] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 046f21098e2344e3b444ad86670ab346 in term 1.
I20260810 03:42:50.478840 5455 leader_election.cc:302] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 046f21098e2344e3b444ad86670ab346, 27dd45a0f7d9402aa766cc714ef0c60d, 4b6e6da672e44e2396fc704ec88bcfb3; no voters:
I20260810 03:42:50.479064 5758 raft_consensus.cc:2781] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:42:50.479300 5758 raft_consensus.cc:686] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [term 1 LEADER]: Becoming Leader. State: Replica: 046f21098e2344e3b444ad86670ab346, State: Running, Role: LEADER
I20260810 03:42:50.479648 5758 consensus_queue.cc:237] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } } peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } }
I20260810 03:42:50.482978 5208 catalog_manager.cc:5495] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 reported cstate change: term changed from 0 to 1, leader changed from <none> to 046f21098e2344e3b444ad86670ab346 (127.21.175.66). New cstate: current_term: 1 leader_uuid: "046f21098e2344e3b444ad86670ab346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } health_report { overall_health: UNKNOWN } } }
W20260810 03:42:50.505656 5740 tablet.cc:2369] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:42:50.507258 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:50.503302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.21.175.66:35821)
0810 03:42:50.503474 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:50.503478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:50.503504 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:42:50.503635 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:50.503643 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:50.503700 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:50.503819 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:42:50.504085 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.504646 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.505577 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:50.505796 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:50.505919 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:50.506287 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:42:50.506766 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:42:50.506864 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:50.507026 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 03:42:50.507107 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 03:42:50.513288 5373 raft_consensus.cc:1261] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Refusing update from remote peer 046f21098e2344e3b444ad86670ab346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:42:50.513293 5683 raft_consensus.cc:1261] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d [term 1 FOLLOWER]: Refusing update from remote peer 046f21098e2344e3b444ad86670ab346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:42:50.514088 5765 consensus_queue.cc:1035] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:42:50.514328 5758 consensus_queue.cc:1035] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:42:50.521225 5770 mvcc.cc:205] Tried to move back new op lower bound from 7316821485615431680 to 7316821485487489024. Current Snapshot: MvccSnapshot[applied={T|T < 7316821485615431680}]
I20260810 03:42:50.522140 5768 mvcc.cc:205] Tried to move back new op lower bound from 7316821485615431680 to 7316821485487489024. Current Snapshot: MvccSnapshot[applied={T|T < 7316821485615431680 or (T in {7316821485615431680})}]
I20260810 03:42:50.524830 5769 mvcc.cc:205] Tried to move back new op lower bound from 7316821485615431680 to 7316821485487489024. Current Snapshot: MvccSnapshot[applied={T|T < 7316821485615431680 or (T in {7316821485615431680})}]
I20260810 03:42:50.529129 5208 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:42:50.529274 5208 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:42:50.531997 5208 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734: Sending DeleteTablet for 3 replicas of tablet 29ddbc910cd24369ad701a35084487a7
I20260810 03:42:50.532878 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 5249
I20260810 03:42:50.532881 5509 tablet_service.cc:1514] Processing DeleteTablet for tablet 29ddbc910cd24369ad701a35084487a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:42:50 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:49972
I20260810 03:42:50.533447 5658 tablet_service.cc:1514] Processing DeleteTablet for tablet 29ddbc910cd24369ad701a35084487a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:42:50 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:33274
I20260810 03:42:50.535332 5780 tablet_replica.cc:330] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d: stopping tablet replica
I20260810 03:42:50.535725 5780 raft_consensus.cc:2229] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:42:50.536091 5780 raft_consensus.cc:2258] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:42:50.536648 5778 tablet_replica.cc:330] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346: stopping tablet replica
I20260810 03:42:50.537035 5778 raft_consensus.cc:2229] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:42:50.537465 5778 raft_consensus.cc:2258] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:42:50.537905 5780 ts_tablet_manager.cc:1905] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:42:50.538910 5778 ts_tablet_manager.cc:1905] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:42:50.541565 5780 ts_tablet_manager.cc:1918] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:42:50.541692 5780 log.cc:1196] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/29ddbc910cd24369ad701a35084487a7
I20260810 03:42:50.541880 5778 ts_tablet_manager.cc:1918] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:42:50.542039 5778 log.cc:1196] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/29ddbc910cd24369ad701a35084487a7
I20260810 03:42:50.542066 5780 ts_tablet_manager.cc:1939] T 29ddbc910cd24369ad701a35084487a7 P 27dd45a0f7d9402aa766cc714ef0c60d: Deleting consensus metadata
I20260810 03:42:50.542469 5778 ts_tablet_manager.cc:1939] T 29ddbc910cd24369ad701a35084487a7 P 046f21098e2344e3b444ad86670ab346: Deleting consensus metadata
I20260810 03:42:50.543613 5192 catalog_manager.cc:4841] TS 27dd45a0f7d9402aa766cc714ef0c60d (127.21.175.67:40255): tablet 29ddbc910cd24369ad701a35084487a7 (table test-table [id=fa9bd0299334437087ec6ca3027432d6]) successfully deleted
I20260810 03:42:50.543632 5190 catalog_manager.cc:4841] TS 046f21098e2344e3b444ad86670ab346 (127.21.175.66:35821): tablet 29ddbc910cd24369ad701a35084487a7 (table test-table [id=fa9bd0299334437087ec6ca3027432d6]) successfully deleted
W20260810 03:42:50.544019 5190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.175.65:35749 (error 108)
I20260810 03:42:50.544639 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 5429
I20260810 03:42:50.547237 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:50.544810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:35749 (local address 127.0.0.1:35016)
0810 03:42:50.547086 (+ 2276us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:50.547160 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2183}
W20260810 03:42:50.548637 5190 catalog_manager.cc:4584] TS 4b6e6da672e44e2396fc704ec88bcfb3 (127.21.175.65:35749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29ddbc910cd24369ad701a35084487a7: Network error: Client connection negotiation failed: client connection to 127.21.175.65:35749: connect: Connection refused (error 111)
I20260810 03:42:50.555321 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 5587
I20260810 03:42:50.563781 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 5167
I20260810 03:42:50.572846 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:36335
--webserver_interface=127.21.175.126
--webserver_port=38097
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:36335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:42:50.689251 5782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:50.689639 5782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:50.689759 5782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:50.693940 5782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:42:50.694005 5782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:50.694024 5782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:50.694041 5782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:42:50.694059 5782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:42:50.699625 5782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:36335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:36335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=38097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:50.700763 5782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:50.701894 5782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:50.710963 5788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:50.710919 5790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:50.711215 5782 server_base.cc:1034] running on GCE node
W20260810 03:42:50.710916 5787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:50.711921 5782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:50.713147 5782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:50.714378 5782 hybrid_clock.cc:648] HybridClock initialized: now 1786333370714296 us; error 89 us; skew 500 ppm
Aug 10 03:42:50 dist-test-slave-ss4n krb5kdc[4526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333370, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:50.717705 5782 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:42:50.718827 5782 webserver.cc:460] Webserver started at http://127.21.175.126:38097/ using document root <none> and password file <none>
I20260810 03:42:50.719480 5782 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:50.719595 5782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:50.724287 5782 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:42:50.726975 5801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:50.728504 5782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:42:50.728675 5782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "4de9004f7e474d2db46d3fa2e8170734"
format_stamp: "Formatted at 2026-08-10 03:42:48 on dist-test-slave-ss4n"
server_key: "bf9f7c3463406d3783f9cb5b8289f152"
server_key_iv: "b07a16796e47e4ed767945e9bd3f16e6"
server_key_version: "encryptionkey@0"
I20260810 03:42:50.729317 5782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:50.747761 5782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:50.751443 5782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:50.751684 5782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:50.760059 5782 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:36335
I20260810 03:42:50.760409 5862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:36335 every 8 connection(s)
I20260810 03:42:50.761375 5782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:42:50.766409 5863 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:42:50.768299 5863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734: Bootstrap starting.
I20260810 03:42:50.770645 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 5782
I20260810 03:42:50.771129 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:35749
--local_ip_for_outbound_sockets=127.21.175.65
--tserver_master_addrs=127.21.175.126:36335
--webserver_port=36477
--webserver_interface=127.21.175.65
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:42:50.794535 5863 log.cc:824] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734: Log is configured to *not* fsync() on all Append() calls
I20260810 03:42:50.810587 5863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:42:50.811118 5863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734: Bootstrap complete.
I20260810 03:42:50.816224 5863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } }
I20260810 03:42:50.816666 5863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de9004f7e474d2db46d3fa2e8170734, State: Initialized, Role: FOLLOWER
I20260810 03:42:50.817302 5863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } }
I20260810 03:42:50.817531 5863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:42:50.817625 5863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:42:50.817760 5863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 2 FOLLOWER]: Advancing to term 3
I20260810 03:42:50.819908 5863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } }
I20260810 03:42:50.820299 5863 leader_election.cc:302] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4de9004f7e474d2db46d3fa2e8170734; no voters:
I20260810 03:42:50.820698 5863 leader_election.cc:288] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 03:42:50.820799 5869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 03:42:50.821043 5869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [term 3 LEADER]: Becoming Leader. State: Replica: 4de9004f7e474d2db46d3fa2e8170734, State: Running, Role: LEADER
I20260810 03:42:50.821372 5869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } }
I20260810 03:42:50.822203 5863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:42:50.822993 5870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4de9004f7e474d2db46d3fa2e8170734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } } }
I20260810 03:42:50.823127 5870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [sys.catalog]: This master's current role is: LEADER
I20260810 03:42:50.824097 5876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:42:50.824220 5871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4de9004f7e474d2db46d3fa2e8170734. Latest consensus state: current_term: 3 leader_uuid: "4de9004f7e474d2db46d3fa2e8170734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9004f7e474d2db46d3fa2e8170734" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 36335 } } }
I20260810 03:42:50.824313 5871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de9004f7e474d2db46d3fa2e8170734 [sys.catalog]: This master's current role is: LEADER
I20260810 03:42:50.829926 5876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:42:50.830823 5876 catalog_manager.cc:1259] Loaded cluster ID: a329cb1f97c446ba8b0a861c6e2e98f6
I20260810 03:42:50.830891 5876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 03:42:50.832182 5876 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc9e10c79ad google::LogMessage::Fail() at ??:0
@ 0x7fc9e10ca88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc9e10c74de google::LogMessage::Flush() at ??:0
@ 0x7fc9e10c8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc9eba0afdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc9eba0bd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc9eba08579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc9eba4c640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc9e33193d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc9e3319c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc9e331b3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc9e330a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc9e494e6db start_thread at ??:0
@ 0x7fc9dfbc561f clone at ??:0
W20260810 03:42:50.933434 5867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:50.933883 5867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:50.933983 5867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:50.940397 5867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:50.940490 5867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:50.940582 5867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:42:50.948696 5867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:35749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=36477
--tserver_master_addrs=127.21.175.126:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:50.950135 5867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:50.951538 5867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:42:50.964058 5867 server_base.cc:1034] running on GCE node
W20260810 03:42:50.964004 5911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:50.964200 5913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:50.964004 5910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:50.964731 5867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:50.965562 5867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:50.966822 5867 hybrid_clock.cc:648] HybridClock initialized: now 1786333370966790 us; error 51 us; skew 500 ppm
Aug 10 03:42:50 dist-test-slave-ss4n krb5kdc[4526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333370, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:50.970114 5867 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:42:50.971138 5867 webserver.cc:460] Webserver started at http://127.21.175.65:36477/ using document root <none> and password file <none>
I20260810 03:42:50.971756 5867 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:50.971825 5867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:50.976253 5867 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:42:50.978916 5921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:50.980409 5867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:42:50.980573 5867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "4b6e6da672e44e2396fc704ec88bcfb3"
format_stamp: "Formatted at 2026-08-10 03:42:48 on dist-test-slave-ss4n"
server_key: "df02fffb25683b22cd43fb159702ac60"
server_key_iv: "9b00ce6b16e604266f0d69bb2f648614"
server_key_version: "encryptionkey@0"
I20260810 03:42:50.981171 5867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:51.000355 5867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:51.004503 5867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:51.004741 5867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:51.005554 5867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:42:51.007054 5929 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 03:42:51.011703 5867 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 03:42:51.011790 5867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260810 03:42:51.011862 5867 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 03:42:51.014142 5867 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 03:42:51.014211 5867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260810 03:42:51.014569 5929 tablet_bootstrap.cc:492] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3: Bootstrap starting.
I20260810 03:42:51.025840 5867 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:35749
I20260810 03:42:51.025892 6052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:35749 every 8 connection(s)
I20260810 03:42:51.027027 5867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:42:51.031311 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 5867
I20260810 03:42:51.031847 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:35821
--local_ip_for_outbound_sockets=127.21.175.66
--tserver_master_addrs=127.21.175.126:36335
--webserver_port=41145
--webserver_interface=127.21.175.66
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:42:51.037235 5929 log.cc:824] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3: Log is configured to *not* fsync() on all Append() calls
I20260810 03:42:51.049139 5929 tablet_bootstrap.cc:492] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:42:51.049679 5929 tablet_bootstrap.cc:492] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3: Bootstrap complete.
I20260810 03:42:51.051297 5929 ts_tablet_manager.cc:1397] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3: Time spent bootstrapping tablet: real 0.037s user 0.030s sys 0.004s
I20260810 03:42:51.054899 5929 raft_consensus.cc:348] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } } peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } }
I20260810 03:42:51.055516 5929 raft_consensus.cc:729] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b6e6da672e44e2396fc704ec88bcfb3, State: Initialized, Role: FOLLOWER
I20260810 03:42:51.056107 5929 consensus_queue.cc:260] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } } peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } }
I20260810 03:42:51.057289 5929 ts_tablet_manager.cc:1428] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260810 03:42:51.077930 6055 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:42:51.029470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:52851)
0810 03:42:51.030189 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:51.030261 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:51.031658 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:51.077055 (+ 45397us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":176,"threads_started":1}
W20260810 03:42:51.078608 6053 heartbeater.cc:644] Failed to heartbeat to 127.21.175.126:36335 (0 consecutive failures): Network error: Failed to ping master at 127.21.175.126:36335: Client connection negotiation failed: client connection to 127.21.175.126:36335: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 03:42:51.079404 6055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:51.079122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:52017)
0810 03:42:51.079276 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:51.079350 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 03:42:51.080186 6055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:51.079916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:57117)
0810 03:42:51.080056 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:51.080120 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 03:42:51.080451 6053 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:42:51.162602 6056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:51.162909 6056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:51.162994 6056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:51.167050 6056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:51.167148 6056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:51.167235 6056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:42:51.172549 6056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:35821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=41145
--tserver_master_addrs=127.21.175.126:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:51.173660 6056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:51.174688 6056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:51.184700 6065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:51.184733 6068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:51.184705 6066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:51.185027 6056 server_base.cc:1034] running on GCE node
I20260810 03:42:51.185623 6056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:51.186321 6056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:51.187546 6056 hybrid_clock.cc:648] HybridClock initialized: now 1786333371187501 us; error 31 us; skew 500 ppm
Aug 10 03:42:51 dist-test-slave-ss4n krb5kdc[4526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333371, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:51.190469 6056 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:42:51.191367 6056 webserver.cc:460] Webserver started at http://127.21.175.66:41145/ using document root <none> and password file <none>
I20260810 03:42:51.191890 6056 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:51.191951 6056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:51.195799 6056 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:42:51.198037 6075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:51.199352 6056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:42:51.199447 6056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "046f21098e2344e3b444ad86670ab346"
format_stamp: "Formatted at 2026-08-10 03:42:48 on dist-test-slave-ss4n"
server_key: "9d38b2b572fd77037a811de2822de28a"
server_key_iv: "e02ecf8d3ce8b28d82e48a72950e7c15"
server_key_version: "encryptionkey@0"
I20260810 03:42:51.200014 6056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:51.226263 6056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:51.230062 6056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:51.230288 6056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:51.230960 6056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:42:51.231818 6056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:42:51.231866 6056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:51.231936 6056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:42:51.231980 6056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:51.240981 6056 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:35821
I20260810 03:42:51.241019 6202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:35821 every 8 connection(s)
I20260810 03:42:51.242092 6056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:42:51.243402 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 6056
I20260810 03:42:51.243914 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:40255
--local_ip_for_outbound_sockets=127.21.175.67
--tserver_master_addrs=127.21.175.126:36335
--webserver_port=36155
--webserver_interface=127.21.175.67
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:42:51.246713 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:51.244656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:41599)
0810 03:42:51.245162 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:51.245298 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":152,"threads_started":1}
W20260810 03:42:51.247231 6203 heartbeater.cc:644] Failed to heartbeat to 127.21.175.126:36335 (0 consecutive failures): Network error: Failed to ping master at 127.21.175.126:36335: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
I20260810 03:42:51.247994 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:51.247675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:53445)
0810 03:42:51.247834 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:51.247906 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 03:42:51.248721 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:51.248418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:43953)
0810 03:42:51.248570 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:51.248635 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 03:42:51.248919 6203 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:42:51.361367 6206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:42:51.361696 6206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:42:51.361740 6206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:42:51.365793 6206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:42:51.365859 6206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:42:51.365921 6206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:42:51.371670 6206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:40255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=36155
--tserver_master_addrs=127.21.175.126:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:42:51.372951 6206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:42:51.374058 6206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:42:51.384564 6216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:51.384536 6211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:42:51.384531 6212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:42:51.384989 6206 server_base.cc:1034] running on GCE node
I20260810 03:42:51.385457 6206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:42:51.386229 6206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:42:51.387428 6206 hybrid_clock.cc:648] HybridClock initialized: now 1786333371387365 us; error 79 us; skew 500 ppm
Aug 10 03:42:51 dist-test-slave-ss4n krb5kdc[4526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333371, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:42:51.390448 6206 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:42:51.391414 6206 webserver.cc:460] Webserver started at http://127.21.175.67:36155/ using document root <none> and password file <none>
I20260810 03:42:51.392040 6206 fs_manager.cc:359] Metadata directory not provided
I20260810 03:42:51.392148 6206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:42:51.396294 6206 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:42:51.398608 6224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:51.399842 6206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 03:42:51.399979 6206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "27dd45a0f7d9402aa766cc714ef0c60d"
format_stamp: "Formatted at 2026-08-10 03:42:49 on dist-test-slave-ss4n"
server_key: "302fb5c905eeb5962c7ce28c33107ebf"
server_key_iv: "f018eb3c40744f81bb5178512541a3a2"
server_key_version: "encryptionkey@0"
I20260810 03:42:51.400586 6206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:42:51.412518 6206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:42:51.416304 6206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:42:51.416476 6206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:42:51.417048 6206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:42:51.417989 6206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:42:51.418037 6206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:51.418079 6206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:42:51.418095 6206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:42:51.427265 6206 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:40255
I20260810 03:42:51.427322 6349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:40255 every 8 connection(s)
I20260810 03:42:51.428347 6206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:42:51.432394 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 6206
I20260810 03:42:51.432698 6355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:51.430735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:41119)
0810 03:42:51.431224 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:51.431346 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":128,"threads_started":1}
W20260810 03:42:51.433202 6352 heartbeater.cc:644] Failed to heartbeat to 127.21.175.126:36335 (0 consecutive failures): Network error: Failed to ping master at 127.21.175.126:36335: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
I20260810 03:42:51.433887 6355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:51.433649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:51835)
0810 03:42:51.433769 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:51.433830 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 03:42:51.434584 6355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:51.434349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:56079)
0810 03:42:51.434466 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:51.434527 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 03:42:51.434772 6352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 03:42:52.081781 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:52.081142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:37675)
0810 03:42:52.081436 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:52.081656 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":78,"threads_started":1}
I20260810 03:42:52.250160 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:52.249695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:49229)
0810 03:42:52.249980 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:52.250046 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":88,"threads_started":1}
I20260810 03:42:52.435833 6362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:52.435387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:35679)
0810 03:42:52.435652 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:52.435711 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260810 03:42:52.643229 6363 raft_consensus.cc:482] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:42:52.643507 6363 raft_consensus.cc:504] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } } peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } }
I20260810 03:42:52.644826 6363 leader_election.cc:288] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 046f21098e2344e3b444ad86670ab346 (127.21.175.66:35821), 27dd45a0f7d9402aa766cc714ef0c60d (127.21.175.67:40255)
Aug 10 03:42:52 dist-test-slave-ss4n krb5kdc[4526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333370, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.67@KRBTEST.COM
Aug 10 03:42:52 dist-test-slave-ss4n krb5kdc[4526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333370, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.66@KRBTEST.COM
I20260810 03:42:52.665163 6365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:52.644989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:35821 (local address 127.21.175.65:48887)
0810 03:42:52.645342 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:52.645359 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:52.645509 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:52.647150 (+ 1641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:52.647153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:52.647710 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:52.648182 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:52.648198 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:52.649521 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:52.649526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:52.650411 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:52.650420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:52.651145 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:52.651149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:52.651412 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:52.657767 (+ 6355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:52.657790 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:52.660072 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:52.662511 (+ 2439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:52.662514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:52.662516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:52.662791 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:52.663189 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:52.663194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:52.663196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:52.663364 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:52.664068 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:52.664076 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:52.664298 (+ 222us) client_negotiation.cc:769] Sending connection context
0810 03:42:52.664493 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 03:42:52.664970 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":88,"spinlock_wait_cycles":9344,"thread_start_us":99,"threads_started":1}
I20260810 03:42:52.665176 6367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:52.645297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:34567 (local address 127.21.175.67:40255)
0810 03:42:52.645716 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:52.645735 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:52.645767 (+ 32us) server_negotiation.cc:407] Connection header received
0810 03:42:52.646433 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:52.646439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:52.646775 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:52.647096 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:52.648376 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:52.649344 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:52.650578 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:52.651023 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:52.651211 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:52.659925 (+ 8714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:52.659934 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:52.659957 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:52.660027 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:52.662310 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:52.662939 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:52.662945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:52.662949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:52.663081 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:52.663488 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:52.663499 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:52.663502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:52.663923 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:52.664096 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:52.664642 (+ 546us) server_negotiation.cc:299] Negotiation successful
0810 03:42:52.664954 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":64,"threads_started":1}
I20260810 03:42:52.665452 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:52.645119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:40255 (local address 127.21.175.65:34567)
0810 03:42:52.645423 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:52.645437 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:42:52.645582 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:42:52.647147 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:42:52.647152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:42:52.647711 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:42:52.648182 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:52.648197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:52.649521 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:52.649525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:52.650414 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:42:52.650424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:42:52.651172 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:42:52.651176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:42:52.651387 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:52.657791 (+ 6404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:42:52.657804 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:42:52.659600 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:42:52.662445 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:52.662451 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:52.662467 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:52.662793 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:52.663210 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:42:52.663212 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:42:52.663214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:42:52.663370 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:42:52.664096 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:42:52.664100 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:42:52.664296 (+ 196us) client_negotiation.cc:769] Sending connection context
0810 03:42:52.664493 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 03:42:52.665171 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":128,"spinlock_wait_cycles":24832,"thread_start_us":64,"threads_started":1}
I20260810 03:42:52.665742 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:52.645192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:48887 (local address 127.21.175.66:35821)
0810 03:42:52.645693 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0810 03:42:52.645727 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0810 03:42:52.645763 (+ 36us) server_negotiation.cc:407] Connection header received
0810 03:42:52.646433 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:42:52.646438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:42:52.646796 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:42:52.647095 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:42:52.648361 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:52.649344 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:52.650538 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:42:52.651012 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:42:52.651201 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:42:52.660281 (+ 9080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:42:52.660286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:42:52.660298 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:42:52.660329 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:42:52.662330 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:52.662932 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:52.662937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:52.662942 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:52.663080 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:42:52.663478 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:42:52.663482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:42:52.663483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:42:52.663922 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:42:52.664070 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 03:42:52.664638 (+ 568us) server_negotiation.cc:299] Negotiation successful
0810 03:42:52.664922 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":118,"threads_started":1}
W20260810 03:42:52.668344 5926 leader_election.cc:341] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 27dd45a0f7d9402aa766cc714ef0c60d (127.21.175.67:40255): Not found: Tablet not found: 29ddbc910cd24369ad701a35084487a7
W20260810 03:42:52.668550 5923 leader_election.cc:341] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 046f21098e2344e3b444ad86670ab346 (127.21.175.66:35821): Not found: Tablet not found: 29ddbc910cd24369ad701a35084487a7
I20260810 03:42:52.668658 5923 leader_election.cc:302] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b6e6da672e44e2396fc704ec88bcfb3; no voters: 046f21098e2344e3b444ad86670ab346, 27dd45a0f7d9402aa766cc714ef0c60d
I20260810 03:42:52.668980 6363 raft_consensus.cc:2726] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:42:53.083022 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:53.082668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:45931)
0810 03:42:53.082851 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:53.082915 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 03:42:53.251428 6368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:53.250989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:37651)
0810 03:42:53.251267 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:53.251336 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":81,"threads_started":1}
I20260810 03:42:53.437139 6370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:53.436679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:48427)
0810 03:42:53.436941 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:53.437002 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":76,"threads_started":1}
I20260810 03:42:54.084585 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:54.083952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:34393)
0810 03:42:54.084328 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:54.084402 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":127,"threads_started":1}
I20260810 03:42:54.252766 6372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:54.252312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:54951)
0810 03:42:54.252592 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:54.252662 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
I20260810 03:42:54.438474 6373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:54.437999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:35873)
0810 03:42:54.438300 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:54.438361 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":81,"threads_started":1}
I20260810 03:42:54.689543 6374 raft_consensus.cc:482] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:42:54.689704 6374 raft_consensus.cc:504] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } } peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } }
I20260810 03:42:54.690276 6374 leader_election.cc:288] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 046f21098e2344e3b444ad86670ab346 (127.21.175.66:35821), 27dd45a0f7d9402aa766cc714ef0c60d (127.21.175.67:40255)
W20260810 03:42:54.691294 5926 leader_election.cc:341] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 27dd45a0f7d9402aa766cc714ef0c60d (127.21.175.67:40255): Not found: Tablet not found: 29ddbc910cd24369ad701a35084487a7
W20260810 03:42:54.691401 5923 leader_election.cc:341] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 046f21098e2344e3b444ad86670ab346 (127.21.175.66:35821): Not found: Tablet not found: 29ddbc910cd24369ad701a35084487a7
I20260810 03:42:54.691501 5923 leader_election.cc:302] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b6e6da672e44e2396fc704ec88bcfb3; no voters: 046f21098e2344e3b444ad86670ab346, 27dd45a0f7d9402aa766cc714ef0c60d
I20260810 03:42:54.691706 6374 raft_consensus.cc:2726] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:42:55.085950 6375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:55.085471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:53087)
0810 03:42:55.085791 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:55.085851 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260810 03:42:55.254271 6376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:55.253820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:46527)
0810 03:42:55.254108 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:55.254170 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260810 03:42:55.440026 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:55.439385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:47493)
0810 03:42:55.439744 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:55.439841 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":95,"threads_started":1}
I20260810 03:42:56.087484 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:56.087006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:38643)
0810 03:42:56.087315 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:56.087386 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":74,"threads_started":1}
I20260810 03:42:56.255748 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:56.255288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:43829)
0810 03:42:56.255584 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:56.255647 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":80,"threads_started":1}
I20260810 03:42:56.441330 6381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:56.440890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:56573)
0810 03:42:56.441174 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:56.441234 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":75,"threads_started":1}
I20260810 03:42:57.071131 6382 raft_consensus.cc:482] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:42:57.071275 6382 raft_consensus.cc:504] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } } peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } }
I20260810 03:42:57.071825 6382 leader_election.cc:288] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 046f21098e2344e3b444ad86670ab346 (127.21.175.66:35821), 27dd45a0f7d9402aa766cc714ef0c60d (127.21.175.67:40255)
W20260810 03:42:57.072630 5923 leader_election.cc:341] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 046f21098e2344e3b444ad86670ab346 (127.21.175.66:35821): Not found: Tablet not found: 29ddbc910cd24369ad701a35084487a7
W20260810 03:42:57.072777 5926 leader_election.cc:341] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 27dd45a0f7d9402aa766cc714ef0c60d (127.21.175.67:40255): Not found: Tablet not found: 29ddbc910cd24369ad701a35084487a7
I20260810 03:42:57.072907 5926 leader_election.cc:302] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b6e6da672e44e2396fc704ec88bcfb3; no voters: 046f21098e2344e3b444ad86670ab346, 27dd45a0f7d9402aa766cc714ef0c60d
I20260810 03:42:57.073151 6382 raft_consensus.cc:2726] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:42:57.088805 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:57.088285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:57907)
0810 03:42:57.088638 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:57.088703 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":119,"threads_started":1}
I20260810 03:42:57.257283 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:57.256662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:57647)
0810 03:42:57.257109 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:57.257180 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":80,"threads_started":1}
I20260810 03:42:57.442884 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:57.442247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:58159)
0810 03:42:57.442710 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:57.442779 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":112,"threads_started":1}
I20260810 03:42:58.090080 6387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:58.089661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:46227)
0810 03:42:58.089926 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:58.089984 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260810 03:42:58.258744 6388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:58.258197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:57375)
0810 03:42:58.258575 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:58.258643 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":81,"threads_started":1}
I20260810 03:42:58.444355 6389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:58.443740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:56495)
0810 03:42:58.444139 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:58.444203 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":121,"threads_started":1}
I20260810 03:42:59.091600 6390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:59.091099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:58743)
0810 03:42:59.091414 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:59.091488 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":98,"threads_started":1}
I20260810 03:42:59.260151 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:59.259671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:37817)
0810 03:42:59.259974 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:59.260047 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":119,"threads_started":1}
I20260810 03:42:59.445803 6392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:42:59.445327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:34575)
0810 03:42:59.445646 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 03:42:59.445705 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":82,"threads_started":1}
I20260810 03:43:00.092856 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:00.092452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:49495)
0810 03:43:00.092697 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:00.092756 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
I20260810 03:43:00.261634 6394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:00.261089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:42789)
0810 03:43:00.261377 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:00.261444 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
I20260810 03:43:00.447058 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:00.446630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:43617)
0810 03:43:00.446896 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:00.446953 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
I20260810 03:43:01.094383 6397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:01.093938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:38537)
0810 03:43:01.094221 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:01.094286 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":80,"threads_started":1}
I20260810 03:43:01.263067 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:01.262601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:33911)
0810 03:43:01.262904 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:01.262969 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":83,"threads_started":1}
I20260810 03:43:01.448402 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:01.447947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:38383)
0810 03:43:01.448243 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:01.448304 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":77,"threads_started":1}
I20260810 03:43:01.813248 6400 raft_consensus.cc:482] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:43:01.813388 6400 raft_consensus.cc:504] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046f21098e2344e3b444ad86670ab346" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 35821 } } peers { permanent_uuid: "27dd45a0f7d9402aa766cc714ef0c60d" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 40255 } } peers { permanent_uuid: "4b6e6da672e44e2396fc704ec88bcfb3" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 35749 } }
I20260810 03:43:01.813961 6400 leader_election.cc:288] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 046f21098e2344e3b444ad86670ab346 (127.21.175.66:35821), 27dd45a0f7d9402aa766cc714ef0c60d (127.21.175.67:40255)
W20260810 03:43:01.814741 5926 leader_election.cc:341] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 27dd45a0f7d9402aa766cc714ef0c60d (127.21.175.67:40255): Not found: Tablet not found: 29ddbc910cd24369ad701a35084487a7
W20260810 03:43:01.814874 5923 leader_election.cc:341] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 046f21098e2344e3b444ad86670ab346 (127.21.175.66:35821): Not found: Tablet not found: 29ddbc910cd24369ad701a35084487a7
I20260810 03:43:01.814963 5923 leader_election.cc:302] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b6e6da672e44e2396fc704ec88bcfb3; no voters: 046f21098e2344e3b444ad86670ab346, 27dd45a0f7d9402aa766cc714ef0c60d
I20260810 03:43:01.815176 6400 raft_consensus.cc:2726] T 29ddbc910cd24369ad701a35084487a7 P 4b6e6da672e44e2396fc704ec88bcfb3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:43:02.095907 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:02.095407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:43831)
0810 03:43:02.095730 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:02.095802 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":110,"threads_started":1}
I20260810 03:43:02.264532 6402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:02.264025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:55371)
0810 03:43:02.264303 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:02.264370 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":86,"threads_started":1}
I20260810 03:43:02.449827 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:02.449363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:42257)
0810 03:43:02.449677 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:02.449734 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":80,"threads_started":1}
I20260810 03:43:03.097289 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:03.096853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:52713)
0810 03:43:03.097133 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:03.097192 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":92,"threads_started":1}
I20260810 03:43:03.265929 6405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:03.265443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:59851)
0810 03:43:03.265753 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:03.265818 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":106,"threads_started":1}
I20260810 03:43:03.451182 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:03.450657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:33369)
0810 03:43:03.450969 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:03.451034 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":97,"threads_started":1}
I20260810 03:43:04.098851 6407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:04.098403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:48487)
0810 03:43:04.098686 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:04.098752 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":84,"threads_started":1}
I20260810 03:43:04.267191 6408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:04.266786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:44453)
0810 03:43:04.267039 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:04.267095 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260810 03:43:04.452499 6409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:04.452044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:41149)
0810 03:43:04.452337 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:04.452397 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":98,"threads_started":1}
I20260810 03:43:05.100212 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:05.099741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:52955)
0810 03:43:05.100032 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:05.100091 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":100,"threads_started":1}
I20260810 03:43:05.268483 6411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:05.268019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:56469)
0810 03:43:05.268321 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:05.268383 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":98,"threads_started":1}
I20260810 03:43:05.453806 6412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:05.453339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:32879)
0810 03:43:05.453642 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:05.453699 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":106,"threads_started":1}
I20260810 03:43:06.101833 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:06.101261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.65:34939)
0810 03:43:06.101659 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:06.101733 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":83,"threads_started":1}
I20260810 03:43:06.270004 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:06.269555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.66:60737)
0810 03:43:06.269844 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:06.269909 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":118,"threads_started":1}
I20260810 03:43:06.444751 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 5867
I20260810 03:43:06.455511 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:06.454893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:36335 (local address 127.21.175.67:47095)
0810 03:43:06.455230 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:06.455350 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":70,"threads_started":1}
I20260810 03:43:06.455626 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 6056
I20260810 03:43:06.470634 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 6206
2026-08-10T03:43:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (20498 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:43:07 dist-test-slave-ss4n krb5kdc[6420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:43:07 dist-test-slave-ss4n krb5kdc[6420](info): set up 2 sockets
Aug 10 03:43:07 dist-test-slave-ss4n krb5kdc[6420](info): commencing operation
krb5kdc: starting...
W20260810 03:43:08.465231 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:43:08 dist-test-slave-ss4n krb5kdc[6420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:43:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:43:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:43:08.608243 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:38079
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:38079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:43:08.726426 6433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:43:08.726791 6433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:43:08.726884 6433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:43:08.731172 6433 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 03:43:08.731258 6433 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 03:43:08.731287 6433 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 03:43:08.731319 6433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:43:08.731365 6433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:43:08.731406 6433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:43:08.731442 6433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:43:08.731664 6433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:43:08.737958 6433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:38079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:38079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:43:08.739209 6433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:43:08.740365 6433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:43:08.751237 6438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:08.751237 6439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:08.751250 6443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:43:08.751497 6433 server_base.cc:1034] running on GCE node
I20260810 03:43:08.752267 6433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:43:08.753525 6433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:43:08.754704 6433 hybrid_clock.cc:648] HybridClock initialized: now 1786333388754670 us; error 48 us; skew 500 ppm
Aug 10 03:43:08 dist-test-slave-ss4n krb5kdc[6420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333388, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:43:08.763168 6433 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:43:08.764209 6433 webserver.cc:460] Webserver started at http://127.21.175.126:33275/ using document root <none> and password file <none>
I20260810 03:43:08.764834 6433 fs_manager.cc:359] Metadata directory not provided
I20260810 03:43:08.764901 6433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:43:08.765125 6433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:43:08.767338 6433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "888ac2a1b607494395e22090b6b124f8"
format_stamp: "Formatted at 2026-08-10 03:43:08 on dist-test-slave-ss4n"
server_key: "3d7c4b51be63d1d25df2735ab61edbd2"
server_key_iv: "c20f7051b47f6d4583107eec0a68c70e"
server_key_version: "encryptionkey@0"
I20260810 03:43:08.767926 6433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "888ac2a1b607494395e22090b6b124f8"
format_stamp: "Formatted at 2026-08-10 03:43:08 on dist-test-slave-ss4n"
server_key: "3d7c4b51be63d1d25df2735ab61edbd2"
server_key_iv: "c20f7051b47f6d4583107eec0a68c70e"
server_key_version: "encryptionkey@0"
I20260810 03:43:08.772292 6433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:43:08.775162 6450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:08.776571 6433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:43:08.776757 6433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "888ac2a1b607494395e22090b6b124f8"
format_stamp: "Formatted at 2026-08-10 03:43:08 on dist-test-slave-ss4n"
server_key: "3d7c4b51be63d1d25df2735ab61edbd2"
server_key_iv: "c20f7051b47f6d4583107eec0a68c70e"
server_key_version: "encryptionkey@0"
I20260810 03:43:08.776880 6433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:43:08.805804 6433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:43:08.809798 6433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:43:08.810040 6433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:43:08.818252 6433 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:38079
I20260810 03:43:08.818596 6511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:38079 every 8 connection(s)
I20260810 03:43:08.819499 6433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:43:08.822480 6512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:43:08.826488 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 6433
I20260810 03:43:08.826591 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:43:08.826872 22205 external_mini_cluster.cc:1428] Setting key 1756617b9449fbf877d859709c34f1f8
I20260810 03:43:08.828691 6512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: Bootstrap starting.
I20260810 03:43:08.831048 6512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: Neither blocks nor log segments found. Creating new log.
I20260810 03:43:08.831856 6512 log.cc:824] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:43:08.833877 6512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: No bootstrap required, opened a new log
Aug 10 03:43:08 dist-test-slave-ss4n krb5kdc[6420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:43:08.838292 6512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } }
I20260810 03:43:08.838529 6512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:43:08.838605 6512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 888ac2a1b607494395e22090b6b124f8, State: Initialized, Role: FOLLOWER
I20260810 03:43:08.839080 6512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } }
I20260810 03:43:08.839241 6512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:43:08.839330 6512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:43:08.839452 6512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:43:08.840658 6512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } }
I20260810 03:43:08.841054 6512 leader_election.cc:302] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 888ac2a1b607494395e22090b6b124f8; no voters:
I20260810 03:43:08.841413 6512 leader_election.cc:288] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:43:08.841523 6518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:43:08.841777 6518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 1 LEADER]: Becoming Leader. State: Replica: 888ac2a1b607494395e22090b6b124f8, State: Running, Role: LEADER
I20260810 03:43:08.842085 6518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } }
I20260810 03:43:08.842864 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:08.828378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40892 (local address 127.21.175.126:38079)
0810 03:43:08.828875 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:08.828890 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:08.828925 (+ 35us) server_negotiation.cc:407] Connection header received
0810 03:43:08.829640 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:08.829659 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:08.829988 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:08.830323 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:43:08.831168 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:08.832250 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:08.833125 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:08.833669 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:08.833835 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:08.836629 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:43:08.836634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:43:08.836648 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:43:08.836676 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:43:08.839029 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:08.839644 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:08.839653 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:08.839663 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:08.839814 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:08.840124 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:08.840127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:08.840129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:08.840733 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:43:08.840941 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:08.841524 (+ 583us) server_negotiation.cc:299] Negotiation successful
0810 03:43:08.841895 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":139,"threads_started":1}
I20260810 03:43:08.842883 6512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:43:08.843693 6518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 888ac2a1b607494395e22090b6b124f8. Latest consensus state: current_term: 1 leader_uuid: "888ac2a1b607494395e22090b6b124f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } } }
I20260810 03:43:08.843693 6520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "888ac2a1b607494395e22090b6b124f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } } }
I20260810 03:43:08.843946 6520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:43:08.843946 6518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [sys.catalog]: This master's current role is: LEADER
W20260810 03:43:08.848244 6535 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:43:08.848317 6535 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:43:08.848404 6524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:43:08.849189 6524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:43:08.855075 6524 catalog_manager.cc:1347] Generated new cluster ID: c6252ef3b2354dca9a2bad7e5521ac58
I20260810 03:43:08.855193 6524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:43:08.878068 6524 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:43:08.879474 6524 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:43:08.888393 6524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: Generated new TSK 0
I20260810 03:43:08.889357 6524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:43:08.928885 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:38079
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:43:09.046182 6542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:43:09.046458 6542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:43:09.046504 6542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:43:09.050580 6542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:43:09.050649 6542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:43:09.050709 6542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:43:09.056098 6542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:43:09.057214 6542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:43:09.058329 6542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:43:09.068761 6550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:09.068907 6548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:43:09.068821 6542 server_base.cc:1034] running on GCE node
W20260810 03:43:09.068773 6552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:43:09.069585 6542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:43:09.070294 6542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:43:09.071509 6542 hybrid_clock.cc:648] HybridClock initialized: now 1786333389071494 us; error 26 us; skew 500 ppm
Aug 10 03:43:09 dist-test-slave-ss4n krb5kdc[6420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:43:09.074651 6542 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:43:09.075549 6542 webserver.cc:460] Webserver started at http://127.21.175.65:39401/ using document root <none> and password file <none>
I20260810 03:43:09.076161 6542 fs_manager.cc:359] Metadata directory not provided
I20260810 03:43:09.076256 6542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:43:09.076499 6542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:43:09.078699 6542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "2972a45571394b35bb66842aff11443d"
format_stamp: "Formatted at 2026-08-10 03:43:09 on dist-test-slave-ss4n"
server_key: "d003ca6450e0b1cde9bc85e034399964"
server_key_iv: "cb4e334327fca6c2f073e96630388399"
server_key_version: "encryptionkey@0"
I20260810 03:43:09.079284 6542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "2972a45571394b35bb66842aff11443d"
format_stamp: "Formatted at 2026-08-10 03:43:09 on dist-test-slave-ss4n"
server_key: "d003ca6450e0b1cde9bc85e034399964"
server_key_iv: "cb4e334327fca6c2f073e96630388399"
server_key_version: "encryptionkey@0"
I20260810 03:43:09.083186 6542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 03:43:09.086006 6559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:09.087390 6542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:43:09.087595 6542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "2972a45571394b35bb66842aff11443d"
format_stamp: "Formatted at 2026-08-10 03:43:09 on dist-test-slave-ss4n"
server_key: "d003ca6450e0b1cde9bc85e034399964"
server_key_iv: "cb4e334327fca6c2f073e96630388399"
server_key_version: "encryptionkey@0"
I20260810 03:43:09.087742 6542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:43:09.117151 6542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:43:09.120746 6542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:43:09.120985 6542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:43:09.121727 6542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:43:09.122623 6542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:43:09.122697 6542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:09.122767 6542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:43:09.122815 6542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:09.131991 6542 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:38089
I20260810 03:43:09.132018 6686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:38089 every 8 connection(s)
I20260810 03:43:09.132936 6542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:43:09.137034 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 6542
I20260810 03:43:09.137161 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:43:09.137571 22205 external_mini_cluster.cc:1428] Setting key fa29e04e7aca9be7c396afca1e13b34e
Aug 10 03:43:09 dist-test-slave-ss4n krb5kdc[6420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:43:09.148964 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.134995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:60893 (local address 127.21.175.126:38079)
0810 03:43:09.135165 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:09.135169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:09.135914 (+ 745us) server_negotiation.cc:407] Connection header received
0810 03:43:09.137611 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:09.137631 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:09.137700 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:09.137788 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:43:09.139272 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.139960 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.141817 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.142062 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.142189 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.144994 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:43:09.145001 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:43:09.145004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:43:09.145032 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:43:09.146737 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:09.147268 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:09.147273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:09.147275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:09.147361 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:09.147722 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:09.147727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:09.147729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:09.147920 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:43:09.148036 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:09.148588 (+ 552us) server_negotiation.cc:299] Negotiation successful
0810 03:43:09.148774 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
I20260810 03:43:09.149830 6689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.135293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:60893)
0810 03:43:09.135743 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:09.135787 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:09.137221 (+ 1434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:09.138028 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:09.138036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:09.138455 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:43:09.139114 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:09.139131 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.140115 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.140120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:09.141665 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:09.141676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.142233 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.142238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:09.142421 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.143130 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:43:09.143147 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:43:09.144818 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:43:09.146876 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:09.146882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:09.146898 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:09.147161 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:09.147472 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:09.147475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:09.147477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:09.147623 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:09.148037 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:43:09.148044 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:43:09.148302 (+ 258us) client_negotiation.cc:769] Sending connection context
0810 03:43:09.148520 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 03:43:09.148892 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":102,"threads_started":1}
I20260810 03:43:09.151268 6687 heartbeater.cc:347] Connected to a master server at 127.21.175.126:38079
I20260810 03:43:09.151532 6687 heartbeater.cc:464] Registering TS with master...
I20260810 03:43:09.152155 6687 heartbeater.cc:511] Master 127.21.175.126:38079 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
I20260810 03:43:09.154181 6472 ts_manager.cc:194] Registered new tserver with Master: 2972a45571394b35bb66842aff11443d (127.21.175.65:38089)
I20260810 03:43:09.155447 6472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:60893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:43:09.168839 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:38079
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:43:09.283728 6692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:43:09.284004 6692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:43:09.284049 6692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:43:09.288323 6692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:43:09.288389 6692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:43:09.288448 6692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:43:09.293959 6692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:43:09.295102 6692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:43:09.296188 6692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:43:09.306543 6697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:09.306558 6701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:09.306531 6698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:43:09.306800 6692 server_base.cc:1034] running on GCE node
I20260810 03:43:09.307332 6692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:43:09.308058 6692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:43:09.309243 6692 hybrid_clock.cc:648] HybridClock initialized: now 1786333389309205 us; error 53 us; skew 500 ppm
Aug 10 03:43:09 dist-test-slave-ss4n krb5kdc[6420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:43:09.312636 6692 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:43:09.313638 6692 webserver.cc:460] Webserver started at http://127.21.175.66:41273/ using document root <none> and password file <none>
I20260810 03:43:09.314226 6692 fs_manager.cc:359] Metadata directory not provided
I20260810 03:43:09.314291 6692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:43:09.314527 6692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:43:09.316673 6692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "98eb3024ec024e4d99b7e138b3ffa542"
format_stamp: "Formatted at 2026-08-10 03:43:09 on dist-test-slave-ss4n"
server_key: "456a1769e2b47a8d23f1bb22b844a97e"
server_key_iv: "6bd98a2ceb1e8858c6c046f45abd3440"
server_key_version: "encryptionkey@0"
I20260810 03:43:09.317279 6692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "98eb3024ec024e4d99b7e138b3ffa542"
format_stamp: "Formatted at 2026-08-10 03:43:09 on dist-test-slave-ss4n"
server_key: "456a1769e2b47a8d23f1bb22b844a97e"
server_key_iv: "6bd98a2ceb1e8858c6c046f45abd3440"
server_key_version: "encryptionkey@0"
I20260810 03:43:09.321446 6692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 03:43:09.324466 6708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:09.326015 6692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:43:09.326192 6692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "98eb3024ec024e4d99b7e138b3ffa542"
format_stamp: "Formatted at 2026-08-10 03:43:09 on dist-test-slave-ss4n"
server_key: "456a1769e2b47a8d23f1bb22b844a97e"
server_key_iv: "6bd98a2ceb1e8858c6c046f45abd3440"
server_key_version: "encryptionkey@0"
I20260810 03:43:09.326414 6692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:43:09.353273 6692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:43:09.356848 6692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:43:09.357080 6692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:43:09.357825 6692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:43:09.358721 6692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:43:09.358769 6692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:09.358810 6692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:43:09.358868 6692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:09.368471 6692 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:42007
I20260810 03:43:09.368507 6838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:42007 every 8 connection(s)
I20260810 03:43:09.369544 6692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:43:09.376781 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 6692
I20260810 03:43:09.376945 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:43:09.377315 22205 external_mini_cluster.cc:1428] Setting key 6f403d43c89e50a709db9108926e8354
Aug 10 03:43:09 dist-test-slave-ss4n krb5kdc[6420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:43:09.385825 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.371584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:40743 (local address 127.21.175.126:38079)
0810 03:43:09.371764 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:09.371767 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:09.373081 (+ 1314us) server_negotiation.cc:407] Connection header received
0810 03:43:09.374538 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:09.374541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:09.374624 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:09.374717 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:43:09.376042 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.376640 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.378344 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.378703 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.378827 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.382059 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:43:09.382063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:43:09.382065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:43:09.382100 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:43:09.383598 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:09.384185 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:09.384189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:09.384190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:09.384265 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:09.384614 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:09.384617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:09.384619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:09.384857 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:43:09.384972 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:09.385515 (+ 543us) server_negotiation.cc:299] Negotiation successful
0810 03:43:09.385651 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 03:43:09.386660 6841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.372266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:40743)
0810 03:43:09.372865 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:09.372939 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:09.374311 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:09.374891 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:09.374898 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:09.375271 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:43:09.375868 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:09.375882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.376825 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.376830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:09.378214 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:09.378225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.378896 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.378901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:09.379147 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.379873 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:43:09.379901 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:43:09.381882 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:43:09.383728 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:09.383734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:09.383752 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:09.384080 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:09.384360 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:09.384364 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:09.384366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:09.384510 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:09.384972 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:43:09.384980 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:43:09.385246 (+ 266us) client_negotiation.cc:769] Sending connection context
0810 03:43:09.385432 (+ 186us) client_negotiation.cc:240] Negotiation successful
0810 03:43:09.385805 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":211,"threads_started":1}
I20260810 03:43:09.387936 6839 heartbeater.cc:347] Connected to a master server at 127.21.175.126:38079
I20260810 03:43:09.388229 6839 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
I20260810 03:43:09.388898 6839 heartbeater.cc:511] Master 127.21.175.126:38079 requested a full tablet report, sending...
I20260810 03:43:09.390195 6472 ts_manager.cc:194] Registered new tserver with Master: 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007)
I20260810 03:43:09.390827 6472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:40743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:43:09.409729 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:38079
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:43:09.525554 6845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:43:09.525849 6845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:43:09.525895 6845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:43:09.530125 6845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:43:09.530193 6845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:43:09.530254 6845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:43:09.535537 6845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:43:09.536686 6845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:43:09.537808 6845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:43:09.548409 6855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:09.548409 6853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:09.548480 6851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:43:09.548861 6845 server_base.cc:1034] running on GCE node
I20260810 03:43:09.549310 6845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:43:09.550091 6845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:43:09.551301 6845 hybrid_clock.cc:648] HybridClock initialized: now 1786333389551281 us; error 47 us; skew 500 ppm
Aug 10 03:43:09 dist-test-slave-ss4n krb5kdc[6420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:43:09.554411 6845 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:43:09.555394 6845 webserver.cc:460] Webserver started at http://127.21.175.67:46575/ using document root <none> and password file <none>
I20260810 03:43:09.556008 6845 fs_manager.cc:359] Metadata directory not provided
I20260810 03:43:09.556149 6845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:43:09.556406 6845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:43:09.558779 6845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "80888a93a2ef4cfc883f7aaaca11c105"
format_stamp: "Formatted at 2026-08-10 03:43:09 on dist-test-slave-ss4n"
server_key: "253abd6a329f94b61aa61c20ae76d9aa"
server_key_iv: "a08e889205648d13de59a905a95e7828"
server_key_version: "encryptionkey@0"
I20260810 03:43:09.559412 6845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "80888a93a2ef4cfc883f7aaaca11c105"
format_stamp: "Formatted at 2026-08-10 03:43:09 on dist-test-slave-ss4n"
server_key: "253abd6a329f94b61aa61c20ae76d9aa"
server_key_iv: "a08e889205648d13de59a905a95e7828"
server_key_version: "encryptionkey@0"
I20260810 03:43:09.563704 6845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:43:09.566624 6862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:09.567991 6845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:43:09.568178 6845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "80888a93a2ef4cfc883f7aaaca11c105"
format_stamp: "Formatted at 2026-08-10 03:43:09 on dist-test-slave-ss4n"
server_key: "253abd6a329f94b61aa61c20ae76d9aa"
server_key_iv: "a08e889205648d13de59a905a95e7828"
server_key_version: "encryptionkey@0"
I20260810 03:43:09.568313 6845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:43:09.598268 6845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:43:09.601389 6845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:43:09.601686 6845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:43:09.602315 6845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:43:09.603165 6845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:43:09.603235 6845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:09.603304 6845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:43:09.603351 6845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:09.612301 6845 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:42825
I20260810 03:43:09.612319 6992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:42825 every 8 connection(s)
I20260810 03:43:09.613266 6845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:43:09.617462 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 6845
I20260810 03:43:09.617702 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:43:09.617995 22205 external_mini_cluster.cc:1428] Setting key 0f10974018b5be9c308c360a845cf380
Aug 10 03:43:09 dist-test-slave-ss4n krb5kdc[6420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:43:09.628970 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.615265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:58567 (local address 127.21.175.126:38079)
0810 03:43:09.615513 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:09.615517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:09.616275 (+ 758us) server_negotiation.cc:407] Connection header received
0810 03:43:09.618153 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:09.618158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:09.618219 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:09.618315 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:43:09.620000 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.620728 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.622046 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.622303 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.622432 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.625207 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:43:09.625211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:43:09.625213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:43:09.625241 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:43:09.626670 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:09.627339 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:09.627344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:09.627345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:09.627433 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:09.627819 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:09.627822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:09.627824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:09.628019 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:43:09.628132 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:09.628659 (+ 527us) server_negotiation.cc:299] Negotiation successful
0810 03:43:09.628791 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260810 03:43:09.629909 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.615585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:58567)
0810 03:43:09.616108 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:09.616158 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:09.617877 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:09.618591 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:09.618600 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:09.619106 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:43:09.619824 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:09.619840 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.620870 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.620875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:09.621900 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:09.621911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.622485 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.622490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:09.622675 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.623269 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:43:09.623288 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:43:09.625008 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:43:09.626811 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:09.626817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:09.626833 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:09.627229 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:09.627532 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:09.627535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:09.627537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:09.627715 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:09.628133 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:43:09.628141 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:43:09.628410 (+ 269us) client_negotiation.cc:769] Sending connection context
0810 03:43:09.628616 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 03:43:09.628960 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":122,"threads_started":1}
I20260810 03:43:09.631233 6993 heartbeater.cc:347] Connected to a master server at 127.21.175.126:38079
I20260810 03:43:09.631531 6993 heartbeater.cc:464] Registering TS with master...
I20260810 03:43:09.632150 6993 heartbeater.cc:511] Master 127.21.175.126:38079 requested a full tablet report, sending...
I20260810 03:43:09.633154 6472 ts_manager.cc:194] Registered new tserver with Master: 80888a93a2ef4cfc883f7aaaca11c105 (127.21.175.67:42825)
I20260810 03:43:09.633724 6472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:58567
I20260810 03:43:09.644549 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:43:09.654776 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.646838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.21.175.126:38079)
0810 03:43:09.647016 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:09.647022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:09.647243 (+ 221us) server_negotiation.cc:407] Connection header received
0810 03:43:09.647327 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:09.647329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:09.647377 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:09.647460 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:43:09.648158 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.648793 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.649680 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.649917 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.650013 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.651541 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:43:09.651544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:43:09.651546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:43:09.651572 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:43:09.653020 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:09.653402 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:09.653405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:09.653406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:09.653569 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:09.653821 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:09.653823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:09.653825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:09.654051 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:43:09.654159 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:09.654386 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 03:43:09.654599 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 03:43:09.657864 6472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:43:09.660888 6472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:43:09.676430 7008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.669840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:42007 (local address 127.0.0.1:50534)
0810 03:43:09.670883 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:09.670937 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:09.671086 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:09.671811 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:09.671821 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:09.671853 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:43:09.672193 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:09.672204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.673669 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.673672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:09.674842 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:09.674851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.675828 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.675832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:09.675952 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.676000 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 03:43:09.676124 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 03:43:09.676255 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"mutex_wait_us":80,"thread_start_us":99,"threads_started":1}
I20260810 03:43:09.676468 7007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.670099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.21.175.66:42007)
0810 03:43:09.670564 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:09.670578 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:09.671289 (+ 711us) server_negotiation.cc:407] Connection header received
0810 03:43:09.671377 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:09.671381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:09.671626 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:09.671790 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:43:09.672382 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.673457 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.674977 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.675696 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.675809 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.675967 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:09.676181 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 03:43:09.676298 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":199,"threads_started":1}
I20260810 03:43:09.676963 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.670267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:42825 (local address 127.0.0.1:45260)
0810 03:43:09.671399 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:09.671415 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:09.671541 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:09.672029 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:09.672032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:09.672052 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:43:09.672242 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:09.672249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.673792 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.673795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:09.675229 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:09.675240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.676496 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.676501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:09.676635 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.676663 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 03:43:09.676739 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 03:43:09.676837 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"mutex_wait_us":92,"thread_start_us":96,"threads_started":1}
I20260810 03:43:09.677528 7010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.670725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45260 (local address 127.21.175.67:42825)
0810 03:43:09.671224 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:09.671239 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:09.671540 (+ 301us) server_negotiation.cc:407] Connection header received
0810 03:43:09.671694 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:09.671702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:09.671893 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:09.672044 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:43:09.672413 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.673617 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.675369 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.676188 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.676291 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.676424 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:09.677205 (+ 781us) server_negotiation.cc:299] Negotiation successful
0810 03:43:09.677332 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":156,"threads_started":1}
I20260810 03:43:09.677595 7006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.670691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:38089 (local address 127.0.0.1:38296)
0810 03:43:09.671775 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:09.671789 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:09.671871 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:09.672611 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:09.672614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:09.672639 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:43:09.672734 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:09.672740 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.674372 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.674376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:09.675556 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:09.675564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.677022 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.677026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:09.677246 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.677270 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:43:09.677345 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 03:43:09.677416 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"thread_start_us":110,"threads_started":1}
I20260810 03:43:09.678045 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.671161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.21.175.65:38089)
0810 03:43:09.671609 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:09.671617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:09.671985 (+ 368us) server_negotiation.cc:407] Connection header received
0810 03:43:09.672076 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:09.672085 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:09.672292 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:09.672443 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:43:09.672879 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.674240 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.675684 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.676634 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.676779 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.676938 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:09.677776 (+ 838us) server_negotiation.cc:299] Negotiation successful
0810 03:43:09.677882 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":99,"threads_started":1}
I20260810 03:43:09.680546 6615 tablet_service.cc:1467] Processing CreateTablet for tablet 2476589f7a1545dabd5260321f37256c (DEFAULT_TABLE table=test-table [id=f3df2e030cf84c6abe27e68e760caf8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:43:09.680428 6916 tablet_service.cc:1467] Processing CreateTablet for tablet 2476589f7a1545dabd5260321f37256c (DEFAULT_TABLE table=test-table [id=f3df2e030cf84c6abe27e68e760caf8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:43:09.680433 6762 tablet_service.cc:1467] Processing CreateTablet for tablet 2476589f7a1545dabd5260321f37256c (DEFAULT_TABLE table=test-table [id=f3df2e030cf84c6abe27e68e760caf8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:43:09.681625 6762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2476589f7a1545dabd5260321f37256c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:43:09.681629 6916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2476589f7a1545dabd5260321f37256c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:43:09.681632 6615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2476589f7a1545dabd5260321f37256c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:43:09.688508 7013 tablet_bootstrap.cc:492] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d: Bootstrap starting.
I20260810 03:43:09.690059 7014 tablet_bootstrap.cc:492] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105: Bootstrap starting.
I20260810 03:43:09.691287 7013 tablet_bootstrap.cc:654] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d: Neither blocks nor log segments found. Creating new log.
I20260810 03:43:09.692406 7013 log.cc:824] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d: Log is configured to *not* fsync() on all Append() calls
I20260810 03:43:09.692571 7014 tablet_bootstrap.cc:654] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105: Neither blocks nor log segments found. Creating new log.
I20260810 03:43:09.693436 7015 tablet_bootstrap.cc:492] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542: Bootstrap starting.
I20260810 03:43:09.694072 7014 log.cc:824] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105: Log is configured to *not* fsync() on all Append() calls
I20260810 03:43:09.695132 7013 tablet_bootstrap.cc:492] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d: No bootstrap required, opened a new log
I20260810 03:43:09.695363 7013 ts_tablet_manager.cc:1397] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 03:43:09.696075 7014 tablet_bootstrap.cc:492] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105: No bootstrap required, opened a new log
I20260810 03:43:09.696293 7014 ts_tablet_manager.cc:1397] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:43:09.696578 7015 tablet_bootstrap.cc:654] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542: Neither blocks nor log segments found. Creating new log.
I20260810 03:43:09.697597 7015 log.cc:824] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542: Log is configured to *not* fsync() on all Append() calls
I20260810 03:43:09.699985 7015 tablet_bootstrap.cc:492] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542: No bootstrap required, opened a new log
I20260810 03:43:09.699929 7014 raft_consensus.cc:348] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:09.700196 7015 ts_tablet_manager.cc:1397] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 03:43:09.700271 7014 raft_consensus.cc:374] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:43:09.700333 7014 raft_consensus.cc:729] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80888a93a2ef4cfc883f7aaaca11c105, State: Initialized, Role: FOLLOWER
I20260810 03:43:09.700538 7013 raft_consensus.cc:348] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:09.700819 7013 raft_consensus.cc:374] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:43:09.700881 7013 raft_consensus.cc:729] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2972a45571394b35bb66842aff11443d, State: Initialized, Role: FOLLOWER
I20260810 03:43:09.700893 7014 consensus_queue.cc:260] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:09.701390 7013 consensus_queue.cc:260] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:09.702118 7014 ts_tablet_manager.cc:1428] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 03:43:09.702239 6687 heartbeater.cc:503] Master 127.21.175.126:38079 was elected leader, sending a full tablet report...
I20260810 03:43:09.702559 6993 heartbeater.cc:503] Master 127.21.175.126:38079 was elected leader, sending a full tablet report...
I20260810 03:43:09.703796 7013 ts_tablet_manager.cc:1428] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260810 03:43:09.704429 7015 raft_consensus.cc:348] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:09.704654 7015 raft_consensus.cc:374] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:43:09.704689 7015 raft_consensus.cc:729] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98eb3024ec024e4d99b7e138b3ffa542, State: Initialized, Role: FOLLOWER
I20260810 03:43:09.705165 7015 consensus_queue.cc:260] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:09.705945 6839 heartbeater.cc:503] Master 127.21.175.126:38079 was elected leader, sending a full tablet report...
I20260810 03:43:09.706265 7015 ts_tablet_manager.cc:1428] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260810 03:43:09.797319 7020 raft_consensus.cc:482] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:43:09.797636 7020 raft_consensus.cc:504] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:09.798906 7020 leader_election.cc:288] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007), 2972a45571394b35bb66842aff11443d (127.21.175.65:38089)
I20260810 03:43:09.804221 7007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.799233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:36433 (local address 127.21.175.66:42007)
0810 03:43:09.799350 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:09.799356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:09.799679 (+ 323us) server_negotiation.cc:407] Connection header received
0810 03:43:09.799744 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:09.799747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:09.799799 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:09.799877 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:43:09.800178 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.800909 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.802885 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.803523 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.803644 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.803701 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:09.804017 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 03:43:09.804092 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 03:43:09.804222 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.799305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:56339 (local address 127.21.175.65:38089)
0810 03:43:09.799431 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:09.799435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:09.799454 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:43:09.799567 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:09.799569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:09.799657 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:09.799752 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:43:09.800098 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.800910 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.802885 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.803509 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.803675 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.803720 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:09.804023 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 03:43:09.804092 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:43:09.804250 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.799147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:38089 (local address 127.21.175.67:56339)
0810 03:43:09.799339 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:09.799355 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:09.799452 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:09.799787 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:09.799789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:09.799803 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:43:09.799948 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:09.799955 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.801062 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.801066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:09.802741 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:09.802753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.803663 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.803668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:09.803838 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.803885 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 03:43:09.803973 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 03:43:09.804080 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":210}
I20260810 03:43:09.804258 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.799149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:42007 (local address 127.21.175.67:36433)
0810 03:43:09.799564 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:09.799577 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:09.799664 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:09.799898 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:09.799901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:09.799920 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:43:09.800029 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:09.800038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.801062 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.801066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:09.802744 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:09.802753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.803663 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.803668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:09.803845 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.803887 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 03:43:09.803985 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 03:43:09.804077 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":278,"thread_start_us":137,"threads_started":1}
I20260810 03:43:09.805224 6786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2476589f7a1545dabd5260321f37256c" candidate_uuid: "80888a93a2ef4cfc883f7aaaca11c105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98eb3024ec024e4d99b7e138b3ffa542" is_pre_election: true
I20260810 03:43:09.805225 6639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2476589f7a1545dabd5260321f37256c" candidate_uuid: "80888a93a2ef4cfc883f7aaaca11c105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2972a45571394b35bb66842aff11443d" is_pre_election: true
I20260810 03:43:09.805590 6786 raft_consensus.cc:2454] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80888a93a2ef4cfc883f7aaaca11c105 in term 0.
I20260810 03:43:09.805590 6639 raft_consensus.cc:2454] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80888a93a2ef4cfc883f7aaaca11c105 in term 0.
I20260810 03:43:09.806118 6865 leader_election.cc:302] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80888a93a2ef4cfc883f7aaaca11c105, 98eb3024ec024e4d99b7e138b3ffa542; no voters:
I20260810 03:43:09.806365 7020 raft_consensus.cc:2781] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:43:09.806454 7020 raft_consensus.cc:482] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:43:09.806486 7020 raft_consensus.cc:3037] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:43:09.807605 7020 raft_consensus.cc:504] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:09.808001 7020 leader_election.cc:288] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [CANDIDATE]: Term 1 election: Requested vote from peers 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007), 2972a45571394b35bb66842aff11443d (127.21.175.65:38089)
I20260810 03:43:09.808403 6786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2476589f7a1545dabd5260321f37256c" candidate_uuid: "80888a93a2ef4cfc883f7aaaca11c105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98eb3024ec024e4d99b7e138b3ffa542"
I20260810 03:43:09.808413 6639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2476589f7a1545dabd5260321f37256c" candidate_uuid: "80888a93a2ef4cfc883f7aaaca11c105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2972a45571394b35bb66842aff11443d"
I20260810 03:43:09.808549 6639 raft_consensus.cc:3037] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:43:09.808544 6786 raft_consensus.cc:3037] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:43:09.809725 6786 raft_consensus.cc:2454] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80888a93a2ef4cfc883f7aaaca11c105 in term 1.
I20260810 03:43:09.809731 6639 raft_consensus.cc:2454] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80888a93a2ef4cfc883f7aaaca11c105 in term 1.
I20260810 03:43:09.810060 6865 leader_election.cc:302] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80888a93a2ef4cfc883f7aaaca11c105, 98eb3024ec024e4d99b7e138b3ffa542; no voters:
I20260810 03:43:09.810245 7020 raft_consensus.cc:2781] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:43:09.810500 7020 raft_consensus.cc:686] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [term 1 LEADER]: Becoming Leader. State: Replica: 80888a93a2ef4cfc883f7aaaca11c105, State: Running, Role: LEADER
I20260810 03:43:09.810902 7020 consensus_queue.cc:237] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:09.814570 6470 catalog_manager.cc:5495] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80888a93a2ef4cfc883f7aaaca11c105 (127.21.175.67). New cstate: current_term: 1 leader_uuid: "80888a93a2ef4cfc883f7aaaca11c105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20260810 03:43:09.832901 7010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.828821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45268 (local address 127.21.175.67:42825)
0810 03:43:09.828943 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:09.828946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:09.828991 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:43:09.829188 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:09.829190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:09.829246 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:09.829327 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:43:09.829609 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.830216 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.831448 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:09.831670 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:09.831753 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:09.832064 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:43:09.832481 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:43:09.832568 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:09.832663 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 03:43:09.832715 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 03:43:09.838898 6639 raft_consensus.cc:1261] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Refusing update from remote peer 80888a93a2ef4cfc883f7aaaca11c105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:43:09.838930 6786 raft_consensus.cc:1261] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542 [term 1 FOLLOWER]: Refusing update from remote peer 80888a93a2ef4cfc883f7aaaca11c105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:43:09.839660 7025 consensus_queue.cc:1035] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:43:09.839861 7020 consensus_queue.cc:1035] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:43:09.850729 7029 mvcc.cc:205] Tried to move back new op lower bound from 7316821564772724736 to 7316821564668215296. Current Snapshot: MvccSnapshot[applied={T|T < 7316821564772724736 or (T in {7316821564772724736})}]
I20260810 03:43:09.856364 6470 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:43:09.856556 6470 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:43:09.859413 6470 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: Sending DeleteTablet for 3 replicas of tablet 2476589f7a1545dabd5260321f37256c
I20260810 03:43:09.860266 6916 tablet_service.cc:1514] Processing DeleteTablet for tablet 2476589f7a1545dabd5260321f37256c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:43:09 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:45260
I20260810 03:43:09.860492 6762 tablet_service.cc:1514] Processing DeleteTablet for tablet 2476589f7a1545dabd5260321f37256c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:43:09 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:50534
I20260810 03:43:09.860507 6615 tablet_service.cc:1514] Processing DeleteTablet for tablet 2476589f7a1545dabd5260321f37256c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:43:09 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:38296
I20260810 03:43:09.861059 7041 tablet_replica.cc:330] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542: stopping tablet replica
I20260810 03:43:09.861235 7040 tablet_replica.cc:330] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105: stopping tablet replica
I20260810 03:43:09.861294 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 6542
I20260810 03:43:09.862499 7040 raft_consensus.cc:2229] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:43:09.862849 7040 raft_consensus.cc:2258] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:43:09.864514 7040 ts_tablet_manager.cc:1905] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:43:09.864813 7041 raft_consensus.cc:2229] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:43:09.865211 7041 raft_consensus.cc:2258] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:43:09.866984 7041 ts_tablet_manager.cc:1905] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:43:09.867527 7040 ts_tablet_manager.cc:1918] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:43:09.867635 7040 log.cc:1196] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/2476589f7a1545dabd5260321f37256c
I20260810 03:43:09.867964 7040 ts_tablet_manager.cc:1939] T 2476589f7a1545dabd5260321f37256c P 80888a93a2ef4cfc883f7aaaca11c105: Deleting consensus metadata
I20260810 03:43:09.869362 6451 catalog_manager.cc:4841] TS 80888a93a2ef4cfc883f7aaaca11c105 (127.21.175.67:42825): tablet 2476589f7a1545dabd5260321f37256c (table test-table [id=f3df2e030cf84c6abe27e68e760caf8a]) successfully deleted
I20260810 03:43:09.870430 7041 ts_tablet_manager.cc:1918] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:43:09.870555 7041 log.cc:1196] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/2476589f7a1545dabd5260321f37256c
I20260810 03:43:09.870903 7041 ts_tablet_manager.cc:1939] T 2476589f7a1545dabd5260321f37256c P 98eb3024ec024e4d99b7e138b3ffa542: Deleting consensus metadata
I20260810 03:43:09.872243 6452 catalog_manager.cc:4841] TS 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007): tablet 2476589f7a1545dabd5260321f37256c (table test-table [id=f3df2e030cf84c6abe27e68e760caf8a]) successfully deleted
W20260810 03:43:09.875000 6455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.175.65:38089 (error 108)
I20260810 03:43:09.875607 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 6692
I20260810 03:43:09.876809 7006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:09.875780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:38089 (local address 127.0.0.1:38302)
0810 03:43:09.876601 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:09.876696 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":705}
W20260810 03:43:09.878067 6455 catalog_manager.cc:4584] TS 2972a45571394b35bb66842aff11443d (127.21.175.65:38089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2476589f7a1545dabd5260321f37256c: Network error: Client connection negotiation failed: client connection to 127.21.175.65:38089: connect: Connection refused (error 111)
I20260810 03:43:09.885273 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 6845
I20260810 03:43:09.891208 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 6433
I20260810 03:43:29.900141 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:38079
--webserver_interface=127.21.175.126
--webserver_port=33275
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:38079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:43:30.022064 7044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:43:30.022377 7044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:43:30.022419 7044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:43:30.026597 7044 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 03:43:30.026657 7044 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 03:43:30.026674 7044 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 03:43:30.026688 7044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:43:30.026706 7044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:43:30.026758 7044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:43:30.026791 7044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:43:30.026818 7044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:43:30.032234 7044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:38079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:38079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=33275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:43:30.033381 7044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:43:30.034554 7044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:43:30.049422 7051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:30.049435 7050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:30.049544 7053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:43:30.049654 7044 server_base.cc:1034] running on GCE node
I20260810 03:43:30.050487 7044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:43:30.051864 7044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:43:30.053086 7044 hybrid_clock.cc:648] HybridClock initialized: now 1786333410053056 us; error 54 us; skew 500 ppm
Aug 10 03:43:30 dist-test-slave-ss4n krb5kdc[6420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333410, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:43:30.063413 7044 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:43:30.064482 7044 webserver.cc:460] Webserver started at http://127.21.175.126:33275/ using document root <none> and password file <none>
I20260810 03:43:30.065116 7044 fs_manager.cc:359] Metadata directory not provided
I20260810 03:43:30.065237 7044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:43:30.069725 7044 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:43:30.072230 7061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:30.073850 7044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:43:30.073998 7044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "888ac2a1b607494395e22090b6b124f8"
format_stamp: "Formatted at 2026-08-10 03:43:08 on dist-test-slave-ss4n"
server_key: "3d7c4b51be63d1d25df2735ab61edbd2"
server_key_iv: "c20f7051b47f6d4583107eec0a68c70e"
server_key_version: "encryptionkey@0"
I20260810 03:43:30.074671 7044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:43:30.094820 7044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:43:30.098095 7044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:43:30.098330 7044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:43:30.106431 7044 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:38079
I20260810 03:43:30.106772 7116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:38079 every 8 connection(s)
I20260810 03:43:30.107694 7044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:43:30.108904 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 7044
I20260810 03:43:30.109357 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:38089
--local_ip_for_outbound_sockets=127.21.175.65
--tserver_master_addrs=127.21.175.126:38079
--webserver_port=39401
--webserver_interface=127.21.175.65
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:43:30.113545 7117 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:43:30.116572 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: Bootstrap starting.
I20260810 03:43:30.143313 7117 log.cc:824] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:43:30.155920 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:43:30.156482 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: Bootstrap complete.
I20260810 03:43:30.163017 7117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } }
I20260810 03:43:30.163599 7117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 888ac2a1b607494395e22090b6b124f8, State: Initialized, Role: FOLLOWER
I20260810 03:43:30.164227 7117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } }
I20260810 03:43:30.164393 7117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:43:30.164480 7117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:43:30.164594 7117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:43:30.167155 7117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } }
I20260810 03:43:30.167565 7117 leader_election.cc:302] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 888ac2a1b607494395e22090b6b124f8; no voters:
I20260810 03:43:30.167948 7117 leader_election.cc:288] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 03:43:30.168092 7121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:43:30.168350 7121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 2 LEADER]: Becoming Leader. State: Replica: 888ac2a1b607494395e22090b6b124f8, State: Running, Role: LEADER
I20260810 03:43:30.168712 7121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } }
I20260810 03:43:30.169301 7117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:43:30.169981 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 888ac2a1b607494395e22090b6b124f8. Latest consensus state: current_term: 2 leader_uuid: "888ac2a1b607494395e22090b6b124f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } } }
I20260810 03:43:30.170107 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:43:30.170522 7127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:43:30.170692 7123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "888ac2a1b607494395e22090b6b124f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } } }
I20260810 03:43:30.170921 7123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:43:30.175928 7127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:43:30.176680 7127 catalog_manager.cc:1259] Loaded cluster ID: c6252ef3b2354dca9a2bad7e5521ac58
I20260810 03:43:30.176741 7127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:43:30.179661 7127 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:43:30.180665 7127 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: Loaded TSK: 0
I20260810 03:43:30.196118 7127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: Generated new TSK 1
I20260810 03:43:30.196424 7127 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:43:30.259733 7119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:43:30.260008 7119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:43:30.260052 7119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:43:30.264240 7119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:43:30.264309 7119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:43:30.264401 7119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:43:30.269935 7119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:38089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=39401
--tserver_master_addrs=127.21.175.126:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:43:30.271348 7119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:43:30.272465 7119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:43:30.283799 7149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:30.283804 7153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:30.283835 7151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:43:30.284255 7119 server_base.cc:1034] running on GCE node
I20260810 03:43:30.284763 7119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:43:30.285688 7119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:43:30.286933 7119 hybrid_clock.cc:648] HybridClock initialized: now 1786333410286903 us; error 49 us; skew 500 ppm
Aug 10 03:43:30 dist-test-slave-ss4n krb5kdc[6420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333410, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:43:30.290047 7119 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:43:30.291078 7119 webserver.cc:460] Webserver started at http://127.21.175.65:39401/ using document root <none> and password file <none>
I20260810 03:43:30.291702 7119 fs_manager.cc:359] Metadata directory not provided
I20260810 03:43:30.291785 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:43:30.296070 7119 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:43:30.299410 7161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:30.300897 7119 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:43:30.301046 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "2972a45571394b35bb66842aff11443d"
format_stamp: "Formatted at 2026-08-10 03:43:09 on dist-test-slave-ss4n"
server_key: "d003ca6450e0b1cde9bc85e034399964"
server_key_iv: "cb4e334327fca6c2f073e96630388399"
server_key_version: "encryptionkey@0"
I20260810 03:43:30.301697 7119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:43:30.314710 7119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:43:30.318387 7119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:43:30.318611 7119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:43:30.319237 7119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:43:30.320425 7169 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 03:43:30.324661 7119 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 03:43:30.324731 7119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260810 03:43:30.324782 7119 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 03:43:30.326532 7119 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 03:43:30.326594 7119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260810 03:43:30.326851 7169 tablet_bootstrap.cc:492] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d: Bootstrap starting.
I20260810 03:43:30.338101 7119 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:38089
I20260810 03:43:30.338112 7290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:38089 every 8 connection(s)
I20260810 03:43:30.339454 7119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:43:30.347972 7169 log.cc:824] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d: Log is configured to *not* fsync() on all Append() calls
I20260810 03:43:30.348999 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 7119
I20260810 03:43:30.349563 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:42007
--local_ip_for_outbound_sockets=127.21.175.66
--tserver_master_addrs=127.21.175.126:38079
--webserver_port=41273
--webserver_interface=127.21.175.66
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:43:30 dist-test-slave-ss4n krb5kdc[6420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333410, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:43:30.357406 7169 tablet_bootstrap.cc:492] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:43:30.357960 7169 tablet_bootstrap.cc:492] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d: Bootstrap complete.
I20260810 03:43:30.359537 7169 ts_tablet_manager.cc:1397] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.004s
I20260810 03:43:30.360015 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.341758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:60013)
0810 03:43:30.342432 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:30.342479 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:30.343479 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:30.344754 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:30.344763 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:30.345312 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:43:30.345907 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:30.345924 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.347475 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.347481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:30.348721 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:30.348734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.349960 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.349965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:30.350169 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.350873 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:43:30.350896 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:43:30.352963 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:43:30.355918 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:30.355924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:30.355937 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:30.356315 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:30.356789 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:30.356792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:30.356795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:30.356982 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:30.357866 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:43:30.357873 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:43:30.358201 (+ 328us) client_negotiation.cc:769] Sending connection context
0810 03:43:30.358443 (+ 242us) client_negotiation.cc:240] Negotiation successful
0810 03:43:30.358860 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":186,"threads_started":1}
I20260810 03:43:30.360062 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.341796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:60013 (local address 127.21.175.126:38079)
0810 03:43:30.342229 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:30.342252 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:30.342666 (+ 414us) server_negotiation.cc:407] Connection header received
0810 03:43:30.343964 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:30.343977 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:30.344314 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:30.344696 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:43:30.346154 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.347312 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.348875 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.349672 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.349879 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.353196 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:43:30.353205 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:43:30.353226 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:43:30.353265 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:43:30.355767 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:30.356460 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:30.356469 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:30.356478 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:30.356663 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:30.357113 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:30.357118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:30.357121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:30.357722 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:43:30.357880 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:30.358622 (+ 742us) server_negotiation.cc:299] Negotiation successful
0810 03:43:30.358985 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":143,"threads_started":1}
I20260810 03:43:30.362628 7169 raft_consensus.cc:348] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:30.363242 7169 raft_consensus.cc:729] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2972a45571394b35bb66842aff11443d, State: Initialized, Role: FOLLOWER
I20260810 03:43:30.363411 7291 heartbeater.cc:347] Connected to a master server at 127.21.175.126:38079
I20260810 03:43:30.363660 7291 heartbeater.cc:464] Registering TS with master...
I20260810 03:43:30.363812 7169 consensus_queue.cc:260] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:30.364207 7291 heartbeater.cc:511] Master 127.21.175.126:38079 requested a full tablet report, sending...
I20260810 03:43:30.364858 7169 ts_tablet_manager.cc:1428] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260810 03:43:30.367108 7077 ts_manager.cc:194] Registered new tserver with Master: 2972a45571394b35bb66842aff11443d (127.21.175.65:38089)
I20260810 03:43:30.371352 7077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:60013
I20260810 03:43:30.374347 7291 heartbeater.cc:503] Master 127.21.175.126:38079 was elected leader, sending a full tablet report...
Aug 10 03:43:30 dist-test-slave-ss4n krb5kdc[6420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333410, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for kudu/127.21.175.65@KRBTEST.COM
I20260810 03:43:30.383509 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.370894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:38089 (local address 127.0.0.1:46122)
0810 03:43:30.371607 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:30.371668 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:30.371966 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:30.372607 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:30.372614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:30.373061 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:43:30.373183 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:30.373192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.374408 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.374412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:30.375543 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:30.375559 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.376384 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.376389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:30.376534 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.377253 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:43:30.377268 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:43:30.379194 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:43:30.381600 (+ 2406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:30.381616 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:30.381620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:30.381878 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:30.382221 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:30.382224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:30.382225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:30.382317 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:30.382874 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:43:30.382880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:43:30.383015 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 03:43:30.383163 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 03:43:30.383351 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":143,"threads_started":1}
I20260810 03:43:30.383615 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.371039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.21.175.65:38089)
0810 03:43:30.371965 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:30.371981 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:30.372013 (+ 32us) server_negotiation.cc:407] Connection header received
0810 03:43:30.372147 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:30.372153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:30.372412 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:30.372607 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:43:30.373343 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.374265 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.375719 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.376261 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.376378 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.379373 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:43:30.379385 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:43:30.379399 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:43:30.379428 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:43:30.381384 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:30.381999 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:30.382006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:30.382008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:30.382106 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:30.382413 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:30.382417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:30.382418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:30.382727 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:43:30.382865 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:30.383265 (+ 400us) server_negotiation.cc:299] Negotiation successful
0810 03:43:30.383441 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":101,"threads_started":1}
I20260810 03:43:30.384727 7214 tablet_service.cc:1514] Processing DeleteTablet for tablet 2476589f7a1545dabd5260321f37256c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:43:09 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:46122
I20260810 03:43:30.385697 7302 tablet_replica.cc:330] stopping tablet replica
I20260810 03:43:30.386001 7302 raft_consensus.cc:2229] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:43:30.386255 7302 raft_consensus.cc:2258] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:43:30.387346 7302 ts_tablet_manager.cc:1905] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:43:30.391382 7302 ts_tablet_manager.cc:1918] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:43:30.391601 7302 log.cc:1196] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal/wals/2476589f7a1545dabd5260321f37256c
I20260810 03:43:30.391949 7302 ts_tablet_manager.cc:1939] T 2476589f7a1545dabd5260321f37256c P 2972a45571394b35bb66842aff11443d: Deleting consensus metadata
I20260810 03:43:30.393808 7065 catalog_manager.cc:4841] TS 2972a45571394b35bb66842aff11443d (127.21.175.65:38089): tablet 2476589f7a1545dabd5260321f37256c (table test-table [id=f3df2e030cf84c6abe27e68e760caf8a]) successfully deleted
W20260810 03:43:30.476385 7298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:43:30.476722 7298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:43:30.476800 7298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:43:30.481346 7298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:43:30.481441 7298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:43:30.481566 7298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:43:30.487175 7298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:42007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=41273
--tserver_master_addrs=127.21.175.126:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:43:30.488344 7298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:43:30.489387 7298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:43:30.499723 7309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:30.499706 7310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:30.499706 7312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:43:30.499801 7298 server_base.cc:1034] running on GCE node
I20260810 03:43:30.500573 7298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:43:30.501304 7298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:43:30.502526 7298 hybrid_clock.cc:648] HybridClock initialized: now 1786333410502501 us; error 44 us; skew 500 ppm
Aug 10 03:43:30 dist-test-slave-ss4n krb5kdc[6420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333410, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:43:30.505785 7298 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:43:30.506791 7298 webserver.cc:460] Webserver started at http://127.21.175.66:41273/ using document root <none> and password file <none>
I20260810 03:43:30.507376 7298 fs_manager.cc:359] Metadata directory not provided
I20260810 03:43:30.507534 7298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:43:30.511812 7298 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 03:43:30.514161 7322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:30.515539 7298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:43:30.515672 7298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "98eb3024ec024e4d99b7e138b3ffa542"
format_stamp: "Formatted at 2026-08-10 03:43:09 on dist-test-slave-ss4n"
server_key: "456a1769e2b47a8d23f1bb22b844a97e"
server_key_iv: "6bd98a2ceb1e8858c6c046f45abd3440"
server_key_version: "encryptionkey@0"
I20260810 03:43:30.516222 7298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:43:30.532764 7298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:43:30.536346 7298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:43:30.536628 7298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:43:30.537302 7298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:43:30.538283 7298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:43:30.538335 7298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:30.538378 7298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:43:30.538393 7298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:30.549029 7298 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:42007
I20260810 03:43:30.549062 7454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:42007 every 8 connection(s)
I20260810 03:43:30.550123 7298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:43:30.558684 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 7298
I20260810 03:43:30.559302 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:42825
--local_ip_for_outbound_sockets=127.21.175.67
--tserver_master_addrs=127.21.175.126:38079
--webserver_port=46575
--webserver_interface=127.21.175.67
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:43:30 dist-test-slave-ss4n krb5kdc[6420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333410, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:43:30.567492 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.552167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:52475 (local address 127.21.175.126:38079)
0810 03:43:30.552317 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:30.552321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:30.553375 (+ 1054us) server_negotiation.cc:407] Connection header received
0810 03:43:30.555074 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:30.555078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:30.555155 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:30.555275 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:43:30.556682 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.557311 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.559061 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.559310 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.559467 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.562990 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:43:30.562997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:43:30.563002 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:43:30.563047 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:43:30.564905 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:30.565474 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:30.565503 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:30.565507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:30.565616 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:30.566090 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:30.566095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:30.566100 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:30.566388 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:43:30.566506 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:30.567069 (+ 563us) server_negotiation.cc:299] Negotiation successful
0810 03:43:30.567244 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:43:30.568151 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.552633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:52475)
0810 03:43:30.553172 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:30.553226 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:30.554816 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:30.555487 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:30.555495 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:30.555927 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:43:30.556489 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:30.556506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.557552 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.557556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:30.558902 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:30.558918 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.559525 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.559534 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:30.559807 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.560543 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:43:30.560575 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:43:30.562781 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:43:30.565053 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:30.565059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:30.565073 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:30.565346 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:30.565813 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:30.565815 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:30.565817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:30.565961 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:30.566502 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:43:30.566508 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:43:30.566789 (+ 281us) client_negotiation.cc:769] Sending connection context
0810 03:43:30.566947 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 03:43:30.567218 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":124,"threads_started":1}
I20260810 03:43:30.569672 7455 heartbeater.cc:347] Connected to a master server at 127.21.175.126:38079
I20260810 03:43:30.569934 7455 heartbeater.cc:464] Registering TS with master...
I20260810 03:43:30.570561 7455 heartbeater.cc:511] Master 127.21.175.126:38079 requested a full tablet report, sending...
I20260810 03:43:30.571806 7077 ts_manager.cc:194] Registered new tserver with Master: 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007)
I20260810 03:43:30.572727 7077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:52475
W20260810 03:43:30.678864 7458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:43:30.679257 7458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:43:30.679306 7458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:43:30.683393 7458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:43:30.683455 7458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:43:30.683516 7458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:43:30.689270 7458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:42825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=46575
--tserver_master_addrs=127.21.175.126:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:43:30.690438 7458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:43:30.691476 7458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:43:30.702036 7464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:30.702023 7468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:43:30.702169 7458 server_base.cc:1034] running on GCE node
W20260810 03:43:30.702031 7465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:43:30.702860 7458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:43:30.703634 7458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:43:30.704867 7458 hybrid_clock.cc:648] HybridClock initialized: now 1786333410704853 us; error 57 us; skew 500 ppm
Aug 10 03:43:30 dist-test-slave-ss4n krb5kdc[6420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333410, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:43:30.708395 7458 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:43:30.709563 7458 webserver.cc:460] Webserver started at http://127.21.175.67:46575/ using document root <none> and password file <none>
I20260810 03:43:30.710209 7458 fs_manager.cc:359] Metadata directory not provided
I20260810 03:43:30.710330 7458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:43:30.714681 7458 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:43:30.716928 7475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:30.718271 7458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 03:43:30.718418 7458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "80888a93a2ef4cfc883f7aaaca11c105"
format_stamp: "Formatted at 2026-08-10 03:43:09 on dist-test-slave-ss4n"
server_key: "253abd6a329f94b61aa61c20ae76d9aa"
server_key_iv: "a08e889205648d13de59a905a95e7828"
server_key_version: "encryptionkey@0"
I20260810 03:43:30.718998 7458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:43:30.739497 7458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:43:30.743391 7458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:43:30.743635 7458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:43:30.744277 7458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:43:30.745237 7458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:43:30.745313 7458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:30.745383 7458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:43:30.745437 7458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:30.755676 7458 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:42825
I20260810 03:43:30.755700 7602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:42825 every 8 connection(s)
I20260810 03:43:30.756732 7458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:43:30.758383 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 7458
Aug 10 03:43:30 dist-test-slave-ss4n krb5kdc[6420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333410, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:43:30.770183 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.760392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.21.175.126:38079)
0810 03:43:30.760690 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:30.760693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:30.760703 (+ 10us) server_negotiation.cc:407] Connection header received
0810 03:43:30.760741 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:30.760744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:30.760782 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:30.760867 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:43:30.761929 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.762546 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.763630 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.763991 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.764065 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.766040 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:43:30.766044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:43:30.766047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:43:30.766077 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:43:30.767943 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:30.768471 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:30.768475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:30.768478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:30.768636 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:30.769035 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:30.769039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:30.769040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:30.769273 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:43:30.769426 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:30.769852 (+ 426us) server_negotiation.cc:299] Negotiation successful
0810 03:43:30.769985 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":247,"thread_start_us":112,"threads_started":1}
I20260810 03:43:30.774084 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.760308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:50583 (local address 127.21.175.126:38079)
0810 03:43:30.760496 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:30.760500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:30.760517 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:43:30.761695 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:30.761698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:30.761741 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:30.761804 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:43:30.763224 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.763816 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.765906 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.766300 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.766412 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.769635 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:43:30.769638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:43:30.769640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:43:30.769783 (+ 143us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:43:30.771258 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:30.771875 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:30.771882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:30.771884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:30.771979 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:30.772597 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:30.772600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:30.772602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:30.772833 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:43:30.772938 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:30.773736 (+ 798us) server_negotiation.cc:299] Negotiation successful
0810 03:43:30.773867 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 03:43:30.775367 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.759229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:50583)
0810 03:43:30.759766 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:30.759812 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:30.761401 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:30.762013 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:30.762021 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:30.762401 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:43:30.763067 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:30.763088 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.763964 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.763967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:30.765749 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:30.765766 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.766551 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.766558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:30.766801 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.767489 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:43:30.767508 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:43:30.769366 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:43:30.771366 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:30.771374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:30.771391 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:30.771719 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:30.772109 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:30.772112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:30.772114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:30.772475 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:30.772944 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:43:30.772950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:43:30.773352 (+ 402us) client_negotiation.cc:769] Sending connection context
0810 03:43:30.773641 (+ 289us) client_negotiation.cc:240] Negotiation successful
0810 03:43:30.774092 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":123,"threads_started":1}
I20260810 03:43:30.779068 7603 heartbeater.cc:347] Connected to a master server at 127.21.175.126:38079
I20260810 03:43:30.779392 7603 heartbeater.cc:464] Registering TS with master...
I20260810 03:43:30.779970 7603 heartbeater.cc:511] Master 127.21.175.126:38079 requested a full tablet report, sending...
I20260810 03:43:30.781075 7077 ts_manager.cc:194] Registered new tserver with Master: 80888a93a2ef4cfc883f7aaaca11c105 (127.21.175.67:42825)
I20260810 03:43:30.781812 7077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:50583
I20260810 03:43:30.790499 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:43:30.795107 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.791217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.21.175.126:38079)
0810 03:43:30.791367 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:30.791371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:30.791611 (+ 240us) server_negotiation.cc:407] Connection header received
0810 03:43:30.791822 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:30.791825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:30.791932 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:30.792057 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:43:30.792344 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.792861 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.793841 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.794073 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.794187 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.794518 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:43:30.794660 (+ 142us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 03:43:30.794955 (+ 295us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51328: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 03:43:30.795198 7293 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51328: token expired
W20260810 03:43:30.795570 7000 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.175.126:38079)
I20260810 03:43:30.805109 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.796013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51340 (local address 127.21.175.126:38079)
0810 03:43:30.796125 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:30.796128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:30.796187 (+ 59us) server_negotiation.cc:407] Connection header received
0810 03:43:30.796317 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:30.796319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:30.796356 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:30.796418 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:43:30.797328 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.797918 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.799269 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.799474 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.799562 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.801262 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:43:30.801265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:43:30.801267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:43:30.801289 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:43:30.803115 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:30.803665 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:30.803667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:30.803669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:30.803764 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:30.804106 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:30.804109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:30.804110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:30.804311 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:43:30.804426 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:30.804808 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 03:43:30.804933 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 03:43:30.808233 7077 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:43:30.811779 7077 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:43:30.821584 7214 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8ca4dc4e564b40bb52a5c85e26aee3 (DEFAULT_TABLE table=test-table [id=3a674cb55a4e4b0a8424dfa329e0a6b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:43:30.822305 7214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8ca4dc4e564b40bb52a5c85e26aee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:43:30.825692 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.820123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:42825 (local address 127.0.0.1:44260)
0810 03:43:30.820363 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:30.820374 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:30.820443 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:30.821658 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:30.821661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:30.821679 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:43:30.821801 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:30.821809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.823046 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.823051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:30.823858 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:30.823867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.825224 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.825227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:30.825317 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.825361 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 03:43:30.825439 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 03:43:30.825545 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":105,"threads_started":1}
I20260810 03:43:30.825752 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.820507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.21.175.67:42825)
0810 03:43:30.821025 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:30.821038 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:30.821061 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:43:30.821153 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:30.821157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:30.821360 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:30.821977 (+ 617us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:43:30.822120 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.822934 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.824317 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.825096 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.825222 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.825379 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:30.825522 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 03:43:30.825618 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":96,"threads_started":1}
I20260810 03:43:30.826440 7169 tablet_bootstrap.cc:492] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d: Bootstrap starting.
I20260810 03:43:30.827433 7169 tablet_bootstrap.cc:654] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d: Neither blocks nor log segments found. Creating new log.
I20260810 03:43:30.829025 7169 tablet_bootstrap.cc:492] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d: No bootstrap required, opened a new log
I20260810 03:43:30.829133 7169 ts_tablet_manager.cc:1397] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:43:30.829595 7169 raft_consensus.cc:348] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:30.829717 7169 raft_consensus.cc:374] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:43:30.829213 7531 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8ca4dc4e564b40bb52a5c85e26aee3 (DEFAULT_TABLE table=test-table [id=3a674cb55a4e4b0a8424dfa329e0a6b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:43:30.829764 7169 raft_consensus.cc:729] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2972a45571394b35bb66842aff11443d, State: Initialized, Role: FOLLOWER
I20260810 03:43:30.829933 7169 consensus_queue.cc:260] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:30.830231 7169 ts_tablet_manager.cc:1428] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:43:30.830399 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8ca4dc4e564b40bb52a5c85e26aee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:43:30.836844 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.819902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:42007 (local address 127.0.0.1:35016)
0810 03:43:30.820095 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:30.820111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:30.820193 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:30.825996 (+ 5803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:30.826003 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:30.826039 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:43:30.826154 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:30.826163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.833630 (+ 7467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.833636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:30.834879 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:30.834895 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.836304 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.836309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:30.836457 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.836497 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 03:43:30.836556 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 03:43:30.836663 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260810 03:43:30.837786 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.820079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35016 (local address 127.21.175.66:42007)
0810 03:43:30.820680 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:30.820692 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:30.820719 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:43:30.820804 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:30.820813 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:30.821109 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:30.821279 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:43:30.826283 (+ 5004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.827340 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.835069 (+ 7729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.836130 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.837183 (+ 1053us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.837380 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:30.837442 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 03:43:30.837589 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":136,"threads_started":1}
W20260810 03:43:30.840708 7292 tablet.cc:2369] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:43:30.841055 7380 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8ca4dc4e564b40bb52a5c85e26aee3 (DEFAULT_TABLE table=test-table [id=3a674cb55a4e4b0a8424dfa329e0a6b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:43:30.841979 7618 tablet_bootstrap.cc:492] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105: Bootstrap starting.
I20260810 03:43:30.842206 7380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8ca4dc4e564b40bb52a5c85e26aee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:43:30.844374 7618 tablet_bootstrap.cc:654] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105: Neither blocks nor log segments found. Creating new log.
I20260810 03:43:30.845764 7618 log.cc:824] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105: Log is configured to *not* fsync() on all Append() calls
I20260810 03:43:30.848074 7618 tablet_bootstrap.cc:492] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105: No bootstrap required, opened a new log
I20260810 03:43:30.848270 7618 ts_tablet_manager.cc:1397] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:43:30.850252 7620 tablet_bootstrap.cc:492] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542: Bootstrap starting.
I20260810 03:43:30.852596 7618 raft_consensus.cc:348] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:30.852830 7620 tablet_bootstrap.cc:654] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542: Neither blocks nor log segments found. Creating new log.
I20260810 03:43:30.852882 7618 raft_consensus.cc:374] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:43:30.852938 7618 raft_consensus.cc:729] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80888a93a2ef4cfc883f7aaaca11c105, State: Initialized, Role: FOLLOWER
I20260810 03:43:30.853412 7618 consensus_queue.cc:260] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:30.853775 7620 log.cc:824] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542: Log is configured to *not* fsync() on all Append() calls
I20260810 03:43:30.854099 7603 heartbeater.cc:503] Master 127.21.175.126:38079 was elected leader, sending a full tablet report...
I20260810 03:43:30.854468 7618 ts_tablet_manager.cc:1428] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 03:43:30.856205 7620 tablet_bootstrap.cc:492] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542: No bootstrap required, opened a new log
I20260810 03:43:30.856426 7620 ts_tablet_manager.cc:1397] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:43:30.860013 7620 raft_consensus.cc:348] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:30.860316 7620 raft_consensus.cc:374] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:43:30.860376 7620 raft_consensus.cc:729] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98eb3024ec024e4d99b7e138b3ffa542, State: Initialized, Role: FOLLOWER
I20260810 03:43:30.860955 7620 consensus_queue.cc:260] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:30.861734 7455 heartbeater.cc:503] Master 127.21.175.126:38079 was elected leader, sending a full tablet report...
I20260810 03:43:30.862068 7620 ts_tablet_manager.cc:1428] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 03:43:30.907557 7299 raft_consensus.cc:482] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:43:30.907881 7299 raft_consensus.cc:504] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:30.909300 7299 leader_election.cc:288] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007), 80888a93a2ef4cfc883f7aaaca11c105 (127.21.175.67:42825)
I20260810 03:43:30.913297 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.909711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:39697 (local address 127.21.175.67:42825)
0810 03:43:30.909852 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:30.909857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:30.910098 (+ 241us) server_negotiation.cc:407] Connection header received
0810 03:43:30.910299 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:30.910303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:30.910354 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:30.910440 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:43:30.910700 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.911210 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.912284 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.912696 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.912830 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.912887 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:30.913092 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 03:43:30.913162 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:43:30.913342 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.909616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:42825 (local address 127.21.175.65:39697)
0810 03:43:30.909994 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:30.910006 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:30.910169 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:30.910457 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:30.910459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:30.910477 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:43:30.910588 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:30.910599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.911327 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.911331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:30.912154 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:30.912163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.912825 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.912829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:30.912916 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.912975 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 03:43:30.913076 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 03:43:30.913158 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":103,"threads_started":1}
I20260810 03:43:30.913522 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.909679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:32833 (local address 127.21.175.66:42007)
0810 03:43:30.909888 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:30.909896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:30.910142 (+ 246us) server_negotiation.cc:407] Connection header received
0810 03:43:30.910332 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:30.910336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:30.910384 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:30.910485 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:43:30.910750 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.911290 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.912403 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.912840 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.913012 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.913068 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:30.913275 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 03:43:30.913355 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:43:30.913679 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.909620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:42007 (local address 127.21.175.65:32833)
0810 03:43:30.910041 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:30.910052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:30.910189 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:30.910498 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:30.910501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:30.910520 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:43:30.910629 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:30.910634 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.911422 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.911426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:30.912264 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:30.912274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.913013 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.913027 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:30.913138 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.913157 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 03:43:30.913258 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 03:43:30.913332 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":48,"thread_start_us":86,"threads_started":1}
I20260810 03:43:30.914222 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8ca4dc4e564b40bb52a5c85e26aee3" candidate_uuid: "2972a45571394b35bb66842aff11443d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80888a93a2ef4cfc883f7aaaca11c105" is_pre_election: true
I20260810 03:43:30.914314 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8ca4dc4e564b40bb52a5c85e26aee3" candidate_uuid: "2972a45571394b35bb66842aff11443d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98eb3024ec024e4d99b7e138b3ffa542" is_pre_election: true
I20260810 03:43:30.914582 7553 raft_consensus.cc:2454] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2972a45571394b35bb66842aff11443d in term 0.
I20260810 03:43:30.914642 7404 raft_consensus.cc:2454] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2972a45571394b35bb66842aff11443d in term 0.
I20260810 03:43:30.915167 7162 leader_election.cc:302] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2972a45571394b35bb66842aff11443d, 80888a93a2ef4cfc883f7aaaca11c105, 98eb3024ec024e4d99b7e138b3ffa542; no voters:
I20260810 03:43:30.915473 7299 raft_consensus.cc:2781] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:43:30.915560 7299 raft_consensus.cc:482] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:43:30.915596 7299 raft_consensus.cc:3037] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:43:30.916872 7299 raft_consensus.cc:504] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:30.917335 7299 leader_election.cc:288] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 1 election: Requested vote from peers 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007), 80888a93a2ef4cfc883f7aaaca11c105 (127.21.175.67:42825)
I20260810 03:43:30.917759 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8ca4dc4e564b40bb52a5c85e26aee3" candidate_uuid: "2972a45571394b35bb66842aff11443d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80888a93a2ef4cfc883f7aaaca11c105"
I20260810 03:43:30.917769 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8ca4dc4e564b40bb52a5c85e26aee3" candidate_uuid: "2972a45571394b35bb66842aff11443d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98eb3024ec024e4d99b7e138b3ffa542"
I20260810 03:43:30.917902 7553 raft_consensus.cc:3037] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:43:30.917918 7404 raft_consensus.cc:3037] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:43:30.919042 7553 raft_consensus.cc:2454] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2972a45571394b35bb66842aff11443d in term 1.
I20260810 03:43:30.919055 7404 raft_consensus.cc:2454] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2972a45571394b35bb66842aff11443d in term 1.
I20260810 03:43:30.919391 7162 leader_election.cc:302] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2972a45571394b35bb66842aff11443d, 80888a93a2ef4cfc883f7aaaca11c105; no voters:
I20260810 03:43:30.919579 7299 raft_consensus.cc:2781] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:43:30.919780 7299 raft_consensus.cc:686] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 LEADER]: Becoming Leader. State: Replica: 2972a45571394b35bb66842aff11443d, State: Running, Role: LEADER
I20260810 03:43:30.920091 7299 consensus_queue.cc:237] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:30.923439 7077 catalog_manager.cc:5495] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2972a45571394b35bb66842aff11443d (127.21.175.65). New cstate: current_term: 1 leader_uuid: "2972a45571394b35bb66842aff11443d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } health_report { overall_health: HEALTHY } } }
I20260810 03:43:30.951176 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.947314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.21.175.65:38089)
0810 03:43:30.947667 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:30.947674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:30.947686 (+ 12us) server_negotiation.cc:407] Connection header received
0810 03:43:30.947764 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:30.947767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:30.947823 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:30.947961 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:43:30.948234 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.948762 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.949668 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:30.949892 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:30.950011 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:30.950307 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:43:30.950762 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:43:30.950895 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:30.950956 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 03:43:30.951013 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":108,"threads_started":1}
I20260810 03:43:30.955227 7553 raft_consensus.cc:1261] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105 [term 1 FOLLOWER]: Refusing update from remote peer 2972a45571394b35bb66842aff11443d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:43:30.955227 7404 raft_consensus.cc:1261] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542 [term 1 FOLLOWER]: Refusing update from remote peer 2972a45571394b35bb66842aff11443d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:43:30.956002 7626 consensus_queue.cc:1035] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:43:30.956247 7299 consensus_queue.cc:1035] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:43:30.964789 7632 mvcc.cc:205] Tried to move back new op lower bound from 7316821651266301952 to 7316821651131179008. Current Snapshot: MvccSnapshot[applied={T|T < 7316821651266301952}]
I20260810 03:43:30.972656 7077 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:43:30.972848 7077 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:43:30.975438 7077 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: Sending DeleteTablet for 3 replicas of tablet 1c8ca4dc4e564b40bb52a5c85e26aee3
I20260810 03:43:30.976194 7214 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8ca4dc4e564b40bb52a5c85e26aee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:43:30 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:46122
I20260810 03:43:30.976171 7380 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8ca4dc4e564b40bb52a5c85e26aee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:43:30 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:35016
I20260810 03:43:30.976171 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8ca4dc4e564b40bb52a5c85e26aee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:43:30 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:44260
I20260810 03:43:30.976701 7645 tablet_replica.cc:330] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542: stopping tablet replica
I20260810 03:43:30.976966 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 7119
I20260810 03:43:30.977016 7645 raft_consensus.cc:2229] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:43:30.979604 7645 raft_consensus.cc:2258] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:43:30.980043 7642 tablet_replica.cc:330] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105: stopping tablet replica
I20260810 03:43:30.980368 7642 raft_consensus.cc:2229] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:43:30.980665 7642 raft_consensus.cc:2258] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:43:30.981745 7645 ts_tablet_manager.cc:1905] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:43:30.983184 7642 ts_tablet_manager.cc:1905] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:43:30.984736 7645 ts_tablet_manager.cc:1918] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:43:30.984845 7645 log.cc:1196] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/1c8ca4dc4e564b40bb52a5c85e26aee3
I20260810 03:43:30.985206 7645 ts_tablet_manager.cc:1939] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 98eb3024ec024e4d99b7e138b3ffa542: Deleting consensus metadata
I20260810 03:43:30.986019 7642 ts_tablet_manager.cc:1918] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:43:30.986129 7642 log.cc:1196] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/1c8ca4dc4e564b40bb52a5c85e26aee3
I20260810 03:43:30.986500 7642 ts_tablet_manager.cc:1939] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 80888a93a2ef4cfc883f7aaaca11c105: Deleting consensus metadata
I20260810 03:43:30.986959 7063 catalog_manager.cc:4841] TS 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007): tablet 1c8ca4dc4e564b40bb52a5c85e26aee3 (table test-table [id=3a674cb55a4e4b0a8424dfa329e0a6b4]) successfully deleted
I20260810 03:43:30.987632 7062 catalog_manager.cc:4841] TS 80888a93a2ef4cfc883f7aaaca11c105 (127.21.175.67:42825): tablet 1c8ca4dc4e564b40bb52a5c85e26aee3 (table test-table [id=3a674cb55a4e4b0a8424dfa329e0a6b4]) successfully deleted
W20260810 03:43:30.988240 7065 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.175.65:38089 (error 108)
I20260810 03:43:30.988636 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 7298
I20260810 03:43:30.989714 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:30.989413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:38089 (local address 127.0.0.1:46150)
0810 03:43:30.989575 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:30.989643 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 03:43:30.989908 7065 catalog_manager.cc:4584] TS 2972a45571394b35bb66842aff11443d (127.21.175.65:38089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c8ca4dc4e564b40bb52a5c85e26aee3: Network error: Client connection negotiation failed: client connection to 127.21.175.65:38089: connect: Connection refused (error 111)
I20260810 03:43:30.998430 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 7458
I20260810 03:43:31.006582 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 7044
I20260810 03:43:51.017071 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:38079
--webserver_interface=127.21.175.126
--webserver_port=33275
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:38079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:43:51.132776 7647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:43:51.133131 7647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:43:51.133214 7647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:43:51.137369 7647 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 03:43:51.137468 7647 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 03:43:51.137494 7647 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 03:43:51.137542 7647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:43:51.137584 7647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:43:51.137653 7647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:43:51.137705 7647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:43:51.137743 7647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:43:51.143450 7647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:38079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:38079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=33275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:43:51.144654 7647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:43:51.145793 7647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:43:51.154886 7647 server_base.cc:1034] running on GCE node
W20260810 03:43:51.154928 7654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:51.155140 7652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:51.155138 7656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:43:51.155787 7647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:43:51.156951 7647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:43:51.158135 7647 hybrid_clock.cc:648] HybridClock initialized: now 1786333431158122 us; error 36 us; skew 500 ppm
Aug 10 03:43:51 dist-test-slave-ss4n krb5kdc[6420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333431, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:43:51.166602 7647 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:43:51.167637 7647 webserver.cc:460] Webserver started at http://127.21.175.126:33275/ using document root <none> and password file <none>
I20260810 03:43:51.168264 7647 fs_manager.cc:359] Metadata directory not provided
I20260810 03:43:51.168358 7647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:43:51.172544 7647 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260810 03:43:51.174937 7665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:51.176252 7647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:43:51.176402 7647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "888ac2a1b607494395e22090b6b124f8"
format_stamp: "Formatted at 2026-08-10 03:43:08 on dist-test-slave-ss4n"
server_key: "3d7c4b51be63d1d25df2735ab61edbd2"
server_key_iv: "c20f7051b47f6d4583107eec0a68c70e"
server_key_version: "encryptionkey@0"
I20260810 03:43:51.176981 7647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:43:51.186393 7647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:43:51.189394 7647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:43:51.189630 7647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:43:51.197284 7647 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:38079
I20260810 03:43:51.197610 7726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:38079 every 8 connection(s)
I20260810 03:43:51.198499 7647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:43:51.203049 7728 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:43:51.204849 7728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: Bootstrap starting.
I20260810 03:43:51.205235 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 7647
I20260810 03:43:51.205755 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:38089
--local_ip_for_outbound_sockets=127.21.175.65
--tserver_master_addrs=127.21.175.126:38079
--webserver_port=39401
--webserver_interface=127.21.175.65
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:43:51.222918 7728 log.cc:824] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:43:51.234586 7728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:43:51.235001 7728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: Bootstrap complete.
I20260810 03:43:51.239236 7728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } }
I20260810 03:43:51.239665 7728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 888ac2a1b607494395e22090b6b124f8, State: Initialized, Role: FOLLOWER
I20260810 03:43:51.240221 7728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } }
I20260810 03:43:51.240367 7728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:43:51.240453 7728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:43:51.240579 7728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 2 FOLLOWER]: Advancing to term 3
I20260810 03:43:51.242677 7728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } }
I20260810 03:43:51.243032 7728 leader_election.cc:302] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 888ac2a1b607494395e22090b6b124f8; no voters:
I20260810 03:43:51.243386 7728 leader_election.cc:288] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 03:43:51.243479 7732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 03:43:51.243697 7732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [term 3 LEADER]: Becoming Leader. State: Replica: 888ac2a1b607494395e22090b6b124f8, State: Running, Role: LEADER
I20260810 03:43:51.244017 7732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } }
I20260810 03:43:51.244751 7728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:43:51.245702 7733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "888ac2a1b607494395e22090b6b124f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } } }
I20260810 03:43:51.245836 7733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:43:51.246182 7734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 888ac2a1b607494395e22090b6b124f8. Latest consensus state: current_term: 3 leader_uuid: "888ac2a1b607494395e22090b6b124f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888ac2a1b607494395e22090b6b124f8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38079 } } }
I20260810 03:43:51.246272 7734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:43:51.250044 7747 catalog_manager.cc:1259] Loaded cluster ID: c6252ef3b2354dca9a2bad7e5521ac58
I20260810 03:43:51.250118 7747 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: loading cluster ID for follower catalog manager: success
I20260810 03:43:51.251976 7747 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 888ac2a1b607494395e22090b6b124f8: acquiring CA information for follower catalog manager: success
F20260810 03:43:51.252956 7747 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f446bd5c9ad google::LogMessage::Fail() at ??:0
@ 0x7f446bd5f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f446bd5c4de google::LogMessage::Flush() at ??:0
@ 0x7f446bd5da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f446ea2018d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f44766c5989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f44766a2023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f44766a254b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f447669b652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f447669a9e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f44766e1541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f446df9f512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f446f5e36db start_thread at ??:0
@ 0x7f446a85a61f clone at ??:0
W20260810 03:43:51.355055 7730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:43:51.355417 7730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:43:51.355499 7730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:43:51.360132 7730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:43:51.360231 7730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:43:51.360322 7730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:43:51.366053 7730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:38089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=39401
--tserver_master_addrs=127.21.175.126:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:43:51.367236 7730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:43:51.368332 7730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:43:51.378612 7772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:51.378652 7774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:51.378611 7776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:43:51.379184 7730 server_base.cc:1034] running on GCE node
I20260810 03:43:51.379709 7730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:43:51.380511 7730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:43:51.381702 7730 hybrid_clock.cc:648] HybridClock initialized: now 1786333431381662 us; error 49 us; skew 500 ppm
Aug 10 03:43:51 dist-test-slave-ss4n krb5kdc[6420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333431, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:43:51.385361 7730 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:43:51.386619 7730 webserver.cc:460] Webserver started at http://127.21.175.65:39401/ using document root <none> and password file <none>
I20260810 03:43:51.387362 7730 fs_manager.cc:359] Metadata directory not provided
I20260810 03:43:51.387454 7730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:43:51.393019 7730 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:43:51.395563 7784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:51.397038 7730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 03:43:51.397187 7730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "2972a45571394b35bb66842aff11443d"
format_stamp: "Formatted at 2026-08-10 03:43:09 on dist-test-slave-ss4n"
server_key: "d003ca6450e0b1cde9bc85e034399964"
server_key_iv: "cb4e334327fca6c2f073e96630388399"
server_key_version: "encryptionkey@0"
I20260810 03:43:51.397867 7730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:43:51.418025 7730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:43:51.421659 7730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:43:51.421872 7730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:43:51.422513 7730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:43:51.423662 7791 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 03:43:51.429643 7730 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 03:43:51.429739 7730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260810 03:43:51.429826 7730 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 03:43:51.431721 7730 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 03:43:51.431849 7730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 03:43:51.432112 7791 tablet_bootstrap.cc:492] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d: Bootstrap starting.
I20260810 03:43:51.443401 7730 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:38089
I20260810 03:43:51.443593 7914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:38089 every 8 connection(s)
I20260810 03:43:51.444573 7730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:43:51.454319 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 7730
I20260810 03:43:51.454839 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:42007
--local_ip_for_outbound_sockets=127.21.175.66
--tserver_master_addrs=127.21.175.126:38079
--webserver_port=41273
--webserver_interface=127.21.175.66
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:43:51.455664 7791 log.cc:824] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d: Log is configured to *not* fsync() on all Append() calls
I20260810 03:43:51.463943 7791 tablet_bootstrap.cc:492] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:43:51.464391 7791 tablet_bootstrap.cc:492] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d: Bootstrap complete.
I20260810 03:43:51.465730 7791 ts_tablet_manager.cc:1397] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.011s
I20260810 03:43:51.468453 7791 raft_consensus.cc:348] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:51.468956 7791 raft_consensus.cc:729] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2972a45571394b35bb66842aff11443d, State: Initialized, Role: FOLLOWER
I20260810 03:43:51.469553 7791 consensus_queue.cc:260] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:51.470753 7791 ts_tablet_manager.cc:1428] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260810 03:43:51.492725 7917 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:43:51.446635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:34793)
0810 03:43:51.447145 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:51.447200 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:51.448079 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:51.491929 (+ 43850us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":152,"threads_started":1}
W20260810 03:43:51.493257 7915 heartbeater.cc:644] Failed to heartbeat to 127.21.175.126:38079 (0 consecutive failures): Network error: Failed to ping master at 127.21.175.126:38079: Client connection negotiation failed: client connection to 127.21.175.126:38079: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 03:43:51.494094 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:51.493830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:58509)
0810 03:43:51.493967 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:51.494032 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 03:43:51.494764 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:51.494549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:38567)
0810 03:43:51.494665 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:51.494714 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 03:43:51.494940 7915 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:43:51.583268 7920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:43:51.583595 7920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:43:51.583673 7920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:43:51.587780 7920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:43:51.587879 7920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:43:51.587978 7920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:43:51.593652 7920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:42007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=41273
--tserver_master_addrs=127.21.175.126:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:43:51.594794 7920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:43:51.595886 7920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:43:51.606293 7928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:51.606359 7929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:51.606359 7931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:43:51.606673 7920 server_base.cc:1034] running on GCE node
I20260810 03:43:51.607153 7920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:43:51.607820 7920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:43:51.609023 7920 hybrid_clock.cc:648] HybridClock initialized: now 1786333431608996 us; error 37 us; skew 500 ppm
Aug 10 03:43:51 dist-test-slave-ss4n krb5kdc[6420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333431, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:43:51.612002 7920 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:43:51.612876 7920 webserver.cc:460] Webserver started at http://127.21.175.66:41273/ using document root <none> and password file <none>
I20260810 03:43:51.613428 7920 fs_manager.cc:359] Metadata directory not provided
I20260810 03:43:51.613549 7920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:43:51.617456 7920 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:43:51.619678 7939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:51.620894 7920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:43:51.621024 7920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "98eb3024ec024e4d99b7e138b3ffa542"
format_stamp: "Formatted at 2026-08-10 03:43:09 on dist-test-slave-ss4n"
server_key: "456a1769e2b47a8d23f1bb22b844a97e"
server_key_iv: "6bd98a2ceb1e8858c6c046f45abd3440"
server_key_version: "encryptionkey@0"
I20260810 03:43:51.621704 7920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:43:51.640911 7920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:43:51.643949 7920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:43:51.644166 7920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:43:51.644778 7920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:43:51.645757 7920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:43:51.645825 7920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:51.645896 7920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:43:51.645944 7920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:51.655357 7920 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:42007
I20260810 03:43:51.655383 8064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:42007 every 8 connection(s)
I20260810 03:43:51.656445 7920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:43:51.660635 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:51.658804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:35079)
0810 03:43:51.659289 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:51.659404 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":109,"threads_started":1}
W20260810 03:43:51.661207 8066 heartbeater.cc:644] Failed to heartbeat to 127.21.175.126:38079 (0 consecutive failures): Network error: Failed to ping master at 127.21.175.126:38079: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
I20260810 03:43:51.661974 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:51.661747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:35005)
0810 03:43:51.661859 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:51.661913 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 03:43:51.662585 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:51.662380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:39935)
0810 03:43:51.662482 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:51.662528 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 03:43:51.662755 8066 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 03:43:51.664250 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 7920
I20260810 03:43:51.664662 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:42825
--local_ip_for_outbound_sockets=127.21.175.67
--tserver_master_addrs=127.21.175.126:38079
--webserver_port=46575
--webserver_interface=127.21.175.67
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:43:51.783164 8070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:43:51.783453 8070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:43:51.783528 8070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:43:51.787680 8070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:43:51.787768 8070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:43:51.787853 8070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:43:51.793185 8070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:42825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=46575
--tserver_master_addrs=127.21.175.126:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:43:51.794292 8070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:43:51.795331 8070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:43:51.805521 8076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:43:51.805531 8079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:43:51.805615 8070 server_base.cc:1034] running on GCE node
W20260810 03:43:51.805517 8077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:43:51.806150 8070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:43:51.806830 8070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:43:51.808010 8070 hybrid_clock.cc:648] HybridClock initialized: now 1786333431807988 us; error 36 us; skew 500 ppm
Aug 10 03:43:51 dist-test-slave-ss4n krb5kdc[6420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333431, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:43:51.810810 8070 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:43:51.811672 8070 webserver.cc:460] Webserver started at http://127.21.175.67:46575/ using document root <none> and password file <none>
I20260810 03:43:51.812237 8070 fs_manager.cc:359] Metadata directory not provided
I20260810 03:43:51.812323 8070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:43:51.816089 8070 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:43:51.818264 8088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:51.819447 8070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:43:51.819572 8070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "80888a93a2ef4cfc883f7aaaca11c105"
format_stamp: "Formatted at 2026-08-10 03:43:09 on dist-test-slave-ss4n"
server_key: "253abd6a329f94b61aa61c20ae76d9aa"
server_key_iv: "a08e889205648d13de59a905a95e7828"
server_key_version: "encryptionkey@0"
I20260810 03:43:51.820122 8070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:43:51.842984 8070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:43:51.846199 8070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:43:51.846405 8070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:43:51.847045 8070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:43:51.847879 8070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:43:51.847946 8070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:51.848016 8070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:43:51.848067 8070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:43:51.856864 8070 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:42825
I20260810 03:43:51.856921 8214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:42825 every 8 connection(s)
I20260810 03:43:51.857985 8070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:43:51.862035 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:51.860180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:55095)
0810 03:43:51.860647 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:51.860757 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
I20260810 03:43:51.862419 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 8070
W20260810 03:43:51.862555 8215 heartbeater.cc:644] Failed to heartbeat to 127.21.175.126:38079 (0 consecutive failures): Network error: Failed to ping master at 127.21.175.126:38079: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
I20260810 03:43:51.863237 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:51.862961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:38931)
0810 03:43:51.863079 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:51.863151 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 03:43:51.863986 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:51.863757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:46725)
0810 03:43:51.863860 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:51.863941 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 03:43:51.864152 8215 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 03:43:52.496335 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:52.495626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:47269)
0810 03:43:52.496055 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:52.496149 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":80,"threads_started":1}
I20260810 03:43:52.663753 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:52.663367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:44953)
0810 03:43:52.663603 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:52.663658 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20260810 03:43:52.865123 8222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:52.864712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:46293)
0810 03:43:52.864959 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:52.865018 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260810 03:43:52.884308 8224 raft_consensus.cc:482] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:43:52.884495 8224 raft_consensus.cc:504] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:52.885614 8224 leader_election.cc:288] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007), 80888a93a2ef4cfc883f7aaaca11c105 (127.21.175.67:42825)
Aug 10 03:43:52 dist-test-slave-ss4n krb5kdc[6420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333431, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.67@KRBTEST.COM
Aug 10 03:43:52 dist-test-slave-ss4n krb5kdc[6420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333431, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.66@KRBTEST.COM
I20260810 03:43:52.900075 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:52.886140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:46039 (local address 127.21.175.67:42825)
0810 03:43:52.886541 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:52.886563 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:52.886596 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:43:52.887392 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:52.887401 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:52.887860 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:52.888197 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:43:52.889329 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:52.890365 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:52.891656 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:52.892116 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:52.892290 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:52.895129 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:43:52.895134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:43:52.895148 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:43:52.895191 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:43:52.896993 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:52.897686 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:52.897695 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:52.897703 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:52.897870 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:52.898297 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:52.898300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:52.898302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:52.898867 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:43:52.898974 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:52.899609 (+ 635us) server_negotiation.cc:299] Negotiation successful
0810 03:43:52.899884 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":104,"threads_started":1}
I20260810 03:43:52.900084 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:52.886069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:60171 (local address 127.21.175.66:42007)
0810 03:43:52.886377 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0810 03:43:52.886393 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:43:52.886441 (+ 48us) server_negotiation.cc:407] Connection header received
0810 03:43:52.887393 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:43:52.887401 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:43:52.887860 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:43:52.888197 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:43:52.889328 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:52.890389 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:52.891576 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:43:52.892122 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:43:52.892284 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:52.895373 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:43:52.895378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:43:52.895390 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:43:52.895419 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:43:52.897441 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:52.898009 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:52.898015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:52.898021 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:52.898190 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:43:52.898504 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:43:52.898506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:43:52.898507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:43:52.898910 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:43:52.899086 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 03:43:52.899601 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 03:43:52.899880 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":103,"threads_started":1}
I20260810 03:43:52.900117 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:52.885852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:42825 (local address 127.21.175.65:46039)
0810 03:43:52.885993 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:52.886011 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:52.886124 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:52.888276 (+ 2152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:52.888281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:52.888655 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:43:52.889148 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:52.889165 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:52.890509 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:52.890513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:52.891457 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:52.891468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:52.892248 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:52.892251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:52.892426 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:52.893088 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:43:52.893115 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:43:52.894907 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:43:52.897123 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:52.897131 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:52.897148 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:52.897559 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:52.897997 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:52.898001 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:52.898002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:52.898190 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:52.898989 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:43:52.898999 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:43:52.899292 (+ 293us) client_negotiation.cc:769] Sending connection context
0810 03:43:52.899483 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 03:43:52.899961 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":235}
I20260810 03:43:52.900074 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:52.885852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:42007 (local address 127.21.175.65:60171)
0810 03:43:52.886336 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:52.886351 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:43:52.886457 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:43:52.888694 (+ 2237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:43:52.888696 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:43:52.888867 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:43:52.889147 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:52.889165 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:52.890538 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:52.890541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:52.891456 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:43:52.891467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:43:52.892480 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:43:52.892484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:43:52.892573 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:43:52.893235 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:43:52.893247 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:43:52.895200 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:43:52.897630 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:52.897636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:52.897638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:52.897895 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:52.898297 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:43:52.898300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:43:52.898302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:43:52.898411 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:43:52.899049 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:43:52.899051 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:43:52.899289 (+ 238us) client_negotiation.cc:769] Sending connection context
0810 03:43:52.899483 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 03:43:52.899874 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":290,"spinlock_wait_cycles":12160,"thread_start_us":117,"threads_started":1}
W20260810 03:43:52.902873 7786 leader_election.cc:341] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007): Not found: Tablet not found: 1c8ca4dc4e564b40bb52a5c85e26aee3
W20260810 03:43:52.903049 7785 leader_election.cc:341] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80888a93a2ef4cfc883f7aaaca11c105 (127.21.175.67:42825): Not found: Tablet not found: 1c8ca4dc4e564b40bb52a5c85e26aee3
I20260810 03:43:52.903153 7785 leader_election.cc:302] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2972a45571394b35bb66842aff11443d; no voters: 80888a93a2ef4cfc883f7aaaca11c105, 98eb3024ec024e4d99b7e138b3ffa542
I20260810 03:43:52.903429 8224 raft_consensus.cc:2726] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:43:53.497713 8228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:53.497143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:60867)
0810 03:43:53.497461 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:53.497599 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":80,"threads_started":1}
I20260810 03:43:53.665159 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:53.664594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:59943)
0810 03:43:53.664911 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:53.664996 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":131,"threads_started":1}
I20260810 03:43:53.866357 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:53.865944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:56017)
0810 03:43:53.866214 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:53.866265 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":94,"threads_started":1}
I20260810 03:43:54.415473 8233 raft_consensus.cc:482] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:43:54.415611 8233 raft_consensus.cc:504] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:54.416150 8233 leader_election.cc:288] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007), 80888a93a2ef4cfc883f7aaaca11c105 (127.21.175.67:42825)
W20260810 03:43:54.417021 7786 leader_election.cc:341] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007): Not found: Tablet not found: 1c8ca4dc4e564b40bb52a5c85e26aee3
W20260810 03:43:54.417145 7785 leader_election.cc:341] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80888a93a2ef4cfc883f7aaaca11c105 (127.21.175.67:42825): Not found: Tablet not found: 1c8ca4dc4e564b40bb52a5c85e26aee3
I20260810 03:43:54.417199 7786 leader_election.cc:302] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2972a45571394b35bb66842aff11443d; no voters: 80888a93a2ef4cfc883f7aaaca11c105, 98eb3024ec024e4d99b7e138b3ffa542
I20260810 03:43:54.417356 8233 raft_consensus.cc:2726] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:43:54.498991 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:54.498544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:39677)
0810 03:43:54.498820 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:54.498883 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":85,"threads_started":1}
I20260810 03:43:54.666458 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:54.666005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:38127)
0810 03:43:54.666296 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:54.666363 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":89,"threads_started":1}
I20260810 03:43:54.867841 8236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:54.867301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:49663)
0810 03:43:54.867682 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:54.867749 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":161,"threads_started":1}
I20260810 03:43:55.500571 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:55.500009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:39779)
0810 03:43:55.500365 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:55.500421 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":113,"threads_started":1}
I20260810 03:43:55.667773 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:55.667317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:48563)
0810 03:43:55.667612 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:55.667674 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":77,"threads_started":1}
I20260810 03:43:55.869158 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:55.868687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:47005)
0810 03:43:55.868980 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:55.869050 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":78,"threads_started":1}
I20260810 03:43:56.317164 8240 raft_consensus.cc:482] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:43:56.317448 8240 raft_consensus.cc:504] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:43:56.318073 8240 leader_election.cc:288] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007), 80888a93a2ef4cfc883f7aaaca11c105 (127.21.175.67:42825)
W20260810 03:43:56.318922 7786 leader_election.cc:341] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007): Not found: Tablet not found: 1c8ca4dc4e564b40bb52a5c85e26aee3
W20260810 03:43:56.319026 7785 leader_election.cc:341] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80888a93a2ef4cfc883f7aaaca11c105 (127.21.175.67:42825): Not found: Tablet not found: 1c8ca4dc4e564b40bb52a5c85e26aee3
I20260810 03:43:56.319150 7786 leader_election.cc:302] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2972a45571394b35bb66842aff11443d; no voters: 80888a93a2ef4cfc883f7aaaca11c105, 98eb3024ec024e4d99b7e138b3ffa542
I20260810 03:43:56.319312 8240 raft_consensus.cc:2726] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:43:56.502072 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:56.501458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:39013)
0810 03:43:56.501901 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:56.501962 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":135,"threads_started":1}
I20260810 03:43:56.669239 8242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:56.668742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:37439)
0810 03:43:56.669065 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:56.669141 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":88,"threads_started":1}
I20260810 03:43:56.870486 8243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:56.870050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:32863)
0810 03:43:56.870337 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:56.870392 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":92,"threads_started":1}
I20260810 03:43:57.503338 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:57.502922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:35067)
0810 03:43:57.503178 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:57.503238 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260810 03:43:57.670758 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:57.670253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:55545)
0810 03:43:57.670560 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:57.670642 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":79,"threads_started":1}
I20260810 03:43:57.871773 8247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:57.871330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:48255)
0810 03:43:57.871612 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:57.871675 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":101,"threads_started":1}
I20260810 03:43:58.504738 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:58.504208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:46851)
0810 03:43:58.504497 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:58.504574 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
I20260810 03:43:58.672103 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:58.671651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:45065)
0810 03:43:58.671949 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:58.672009 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":74,"threads_started":1}
I20260810 03:43:58.873139 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:58.872579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:43217)
0810 03:43:58.872965 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:58.873033 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":75,"threads_started":1}
I20260810 03:43:59.506009 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:59.505546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:50295)
0810 03:43:59.505830 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:59.505903 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":80,"threads_started":1}
I20260810 03:43:59.673719 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:59.673127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:58017)
0810 03:43:59.673416 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:59.673612 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
I20260810 03:43:59.874481 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:43:59.874041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:40983)
0810 03:43:59.874319 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 03:43:59.874383 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260810 03:44:00.507367 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:00.506906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:59889)
0810 03:44:00.507206 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:00.507270 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":84,"threads_started":1}
I20260810 03:44:00.675024 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:00.674591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:50701)
0810 03:44:00.674872 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:00.674928 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":84,"threads_started":1}
I20260810 03:44:00.875747 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:00.875253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:57981)
0810 03:44:00.875588 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:00.875650 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":98,"threads_started":1}
I20260810 03:44:00.958638 8258 raft_consensus.cc:482] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:44:00.958786 8258 raft_consensus.cc:504] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:44:00.959354 8258 leader_election.cc:288] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007), 80888a93a2ef4cfc883f7aaaca11c105 (127.21.175.67:42825)
W20260810 03:44:00.960320 7785 leader_election.cc:341] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80888a93a2ef4cfc883f7aaaca11c105 (127.21.175.67:42825): Not found: Tablet not found: 1c8ca4dc4e564b40bb52a5c85e26aee3
W20260810 03:44:00.960552 7786 leader_election.cc:341] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007): Not found: Tablet not found: 1c8ca4dc4e564b40bb52a5c85e26aee3
I20260810 03:44:00.960633 7786 leader_election.cc:302] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2972a45571394b35bb66842aff11443d; no voters: 80888a93a2ef4cfc883f7aaaca11c105, 98eb3024ec024e4d99b7e138b3ffa542
I20260810 03:44:00.960806 8258 raft_consensus.cc:2726] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:44:01.508905 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:01.508420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:35043)
0810 03:44:01.508735 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:01.508806 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":98,"threads_started":1}
I20260810 03:44:01.676573 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:01.676024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:57383)
0810 03:44:01.676343 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:01.676410 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
I20260810 03:44:01.877105 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:01.876624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:47899)
0810 03:44:01.876937 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:01.877004 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":81,"threads_started":1}
I20260810 03:44:02.510511 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:02.510019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:51295)
0810 03:44:02.510335 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:02.510405 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":98,"threads_started":1}
I20260810 03:44:02.678251 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:02.677749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:49405)
0810 03:44:02.678071 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:02.678152 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
I20260810 03:44:02.878454 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:02.878021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:34247)
0810 03:44:02.878287 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:02.878356 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
I20260810 03:44:03.512089 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:03.511574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:57871)
0810 03:44:03.511909 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:03.511975 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":125,"threads_started":1}
I20260810 03:44:03.679628 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:03.679211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:35957)
0810 03:44:03.679472 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:03.679529 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":81,"threads_started":1}
I20260810 03:44:03.879745 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:03.879262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:41617)
0810 03:44:03.879541 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:03.879598 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":98,"threads_started":1}
I20260810 03:44:03.899396 8268 raft_consensus.cc:482] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:44:03.899550 8268 raft_consensus.cc:504] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98eb3024ec024e4d99b7e138b3ffa542" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 42007 } } peers { permanent_uuid: "80888a93a2ef4cfc883f7aaaca11c105" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 42825 } } peers { permanent_uuid: "2972a45571394b35bb66842aff11443d" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 38089 } }
I20260810 03:44:03.900063 8268 leader_election.cc:288] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007), 80888a93a2ef4cfc883f7aaaca11c105 (127.21.175.67:42825)
W20260810 03:44:03.900921 7786 leader_election.cc:341] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98eb3024ec024e4d99b7e138b3ffa542 (127.21.175.66:42007): Not found: Tablet not found: 1c8ca4dc4e564b40bb52a5c85e26aee3
W20260810 03:44:03.901070 7785 leader_election.cc:341] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80888a93a2ef4cfc883f7aaaca11c105 (127.21.175.67:42825): Not found: Tablet not found: 1c8ca4dc4e564b40bb52a5c85e26aee3
I20260810 03:44:03.901171 7785 leader_election.cc:302] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2972a45571394b35bb66842aff11443d; no voters: 80888a93a2ef4cfc883f7aaaca11c105, 98eb3024ec024e4d99b7e138b3ffa542
I20260810 03:44:03.901306 8268 raft_consensus.cc:2726] T 1c8ca4dc4e564b40bb52a5c85e26aee3 P 2972a45571394b35bb66842aff11443d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:44:04.513706 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:04.512948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:34545)
0810 03:44:04.513300 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:04.513374 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":100,"threads_started":1}
I20260810 03:44:04.680895 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:04.680445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:50765)
0810 03:44:04.680733 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:04.680791 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":92,"threads_started":1}
I20260810 03:44:04.881017 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:04.880582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:43083)
0810 03:44:04.880851 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:04.880916 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
I20260810 03:44:05.515096 8273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:05.514626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:53701)
0810 03:44:05.514922 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:05.514986 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":80,"threads_started":1}
I20260810 03:44:05.682469 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:05.681898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:57861)
0810 03:44:05.682247 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:05.682319 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
I20260810 03:44:05.882441 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:05.881980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:60131)
0810 03:44:05.882277 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:05.882341 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":98,"threads_started":1}
I20260810 03:44:06.516685 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:06.516128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.65:35431)
0810 03:44:06.516431 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:06.516533 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":82,"threads_started":1}
I20260810 03:44:06.683957 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:06.683489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.66:56923)
0810 03:44:06.683782 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:06.683851 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":75,"threads_started":1}
I20260810 03:44:06.868999 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 7730
I20260810 03:44:06.877839 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 7920
I20260810 03:44:06.883961 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:06.883474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38079 (local address 127.21.175.67:43307)
0810 03:44:06.883773 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:06.883838 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.126:38079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":97,"threads_started":1}
I20260810 03:44:06.886478 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 8070
2026-08-10T03:44:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59570 ms)
[----------] 30 tests from SecurityITest (256849 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:06 dist-test-slave-ss4n krb5kdc[8282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:06 dist-test-slave-ss4n krb5kdc[8282](info): set up 2 sockets
Aug 10 03:44:06 dist-test-slave-ss4n krb5kdc[8282](info): commencing operation
krb5kdc: starting...
W20260810 03:44:08.014763 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:08 dist-test-slave-ss4n krb5kdc[8282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:44:08.152243 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:40377
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:40377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:08.267948 8296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:08.268280 8296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:08.268359 8296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:08.272454 8296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:08.272547 8296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:08.272583 8296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:08.272626 8296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:08.272660 8296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:08.278120 8296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:40377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:40377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:08.279635 8296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:08.280987 8296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:08.290411 8304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:08.290488 8302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:08.290529 8296 server_base.cc:1034] running on GCE node
W20260810 03:44:08.290580 8301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:08.291443 8296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:08.292730 8296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:08.293926 8296 hybrid_clock.cc:648] HybridClock initialized: now 1786333448293878 us; error 59 us; skew 500 ppm
Aug 10 03:44:08 dist-test-slave-ss4n krb5kdc[8282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333448, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:08.296994 8296 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:44:08.298072 8296 webserver.cc:460] Webserver started at http://127.21.175.126:41305/ using document root <none> and password file <none>
I20260810 03:44:08.298686 8296 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:08.298784 8296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:08.299031 8296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:08.301203 8296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "8c4ac41eca45464b9713ac97eecd7b22"
format_stamp: "Formatted at 2026-08-10 03:44:08 on dist-test-slave-ss4n"
server_key: "2194a5d22661331725d6bc80a12adfc9"
server_key_iv: "f60e07989065e753ea71db6d85736dd6"
server_key_version: "encryptionkey@0"
I20260810 03:44:08.301967 8296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "8c4ac41eca45464b9713ac97eecd7b22"
format_stamp: "Formatted at 2026-08-10 03:44:08 on dist-test-slave-ss4n"
server_key: "2194a5d22661331725d6bc80a12adfc9"
server_key_iv: "f60e07989065e753ea71db6d85736dd6"
server_key_version: "encryptionkey@0"
I20260810 03:44:08.306105 8296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:44:08.309067 8312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:08.310585 8296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:44:08.310732 8296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "8c4ac41eca45464b9713ac97eecd7b22"
format_stamp: "Formatted at 2026-08-10 03:44:08 on dist-test-slave-ss4n"
server_key: "2194a5d22661331725d6bc80a12adfc9"
server_key_iv: "f60e07989065e753ea71db6d85736dd6"
server_key_version: "encryptionkey@0"
I20260810 03:44:08.310877 8296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:08.334558 8296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:08.337913 8296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:08.338080 8296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:08.345395 8296 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:40377
I20260810 03:44:08.345412 8368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:40377 every 8 connection(s)
I20260810 03:44:08.346498 8296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:08.349788 8370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:08.350382 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 8296
I20260810 03:44:08.350533 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:08.350826 22205 external_mini_cluster.cc:1428] Setting key 0bbe8ff80c4b193d0ffc96aa8b00f5e3
I20260810 03:44:08.357939 8370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22: Bootstrap starting.
Aug 10 03:44:08 dist-test-slave-ss4n krb5kdc[8282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:08.360661 8370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:08.361704 8370 log.cc:824] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:08.363932 8370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22: No bootstrap required, opened a new log
I20260810 03:44:08.366010 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:08.352188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52354 (local address 127.21.175.126:40377)
0810 03:44:08.352637 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:08.352653 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:08.352690 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:44:08.353777 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:08.353798 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:08.354170 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:08.354529 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:08.355382 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:08.356447 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:08.357349 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:08.357916 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:08.358121 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:08.360834 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:08.360841 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:08.360863 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:08.360896 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:08.362974 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:08.363467 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:08.363473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:08.363478 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:08.363611 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:08.363902 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:08.363906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:08.363907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:08.364387 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:08.364583 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:08.364905 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 03:44:08.365213 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":143,"threads_started":1}
I20260810 03:44:08.368481 8370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4ac41eca45464b9713ac97eecd7b22" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 40377 } }
I20260810 03:44:08.368835 8370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:08.368916 8370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c4ac41eca45464b9713ac97eecd7b22, State: Initialized, Role: FOLLOWER
I20260810 03:44:08.369431 8370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4ac41eca45464b9713ac97eecd7b22" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 40377 } }
I20260810 03:44:08.369649 8370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:08.369705 8370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:08.369784 8370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:08.370771 8370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4ac41eca45464b9713ac97eecd7b22" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 40377 } }
I20260810 03:44:08.371109 8370 leader_election.cc:302] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c4ac41eca45464b9713ac97eecd7b22; no voters:
I20260810 03:44:08.371397 8370 leader_election.cc:288] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:08.371526 8376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:08.371809 8376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [term 1 LEADER]: Becoming Leader. State: Replica: 8c4ac41eca45464b9713ac97eecd7b22, State: Running, Role: LEADER
I20260810 03:44:08.372180 8376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4ac41eca45464b9713ac97eecd7b22" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 40377 } }
I20260810 03:44:08.372702 8370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:08.374161 8377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c4ac41eca45464b9713ac97eecd7b22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4ac41eca45464b9713ac97eecd7b22" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 40377 } } }
I20260810 03:44:08.374312 8377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:08.374169 8379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c4ac41eca45464b9713ac97eecd7b22. Latest consensus state: current_term: 1 leader_uuid: "8c4ac41eca45464b9713ac97eecd7b22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4ac41eca45464b9713ac97eecd7b22" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 40377 } } }
I20260810 03:44:08.374521 8379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:08.374881 8386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:08.378309 8386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:08.383994 8386 catalog_manager.cc:1347] Generated new cluster ID: 261924a567204bc3bb1faec494161799
I20260810 03:44:08.384073 8386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:08.409093 8386 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:08.410524 8386 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:08.416975 8386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22: Generated new TSK 0
I20260810 03:44:08.417871 8386 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:44:08.452713 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:40377
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:08.569411 8401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:08.569772 8401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:08.569859 8401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:08.574169 8401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:08.574258 8401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:08.574366 8401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:44:08.580042 8401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:40377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:08.581154 8401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:08.582284 8401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:08.592914 8407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:08.592904 8410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:08.593055 8401 server_base.cc:1034] running on GCE node
W20260810 03:44:08.593115 8406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:08.593673 8401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:08.594378 8401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:08.595577 8401 hybrid_clock.cc:648] HybridClock initialized: now 1786333448595534 us; error 53 us; skew 500 ppm
Aug 10 03:44:08 dist-test-slave-ss4n krb5kdc[8282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333448, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:08.598724 8401 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:44:08.599664 8401 webserver.cc:460] Webserver started at http://127.21.175.65:44195/ using document root <none> and password file <none>
I20260810 03:44:08.600255 8401 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:08.600323 8401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:08.600544 8401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:08.602690 8401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "f1e6642ee8ab4469a223efb7ea2f5cd1"
format_stamp: "Formatted at 2026-08-10 03:44:08 on dist-test-slave-ss4n"
server_key: "cf4c328f2924ae7c851348ba620d9c3a"
server_key_iv: "0074872278d7242de41ff5971b76c0d7"
server_key_version: "encryptionkey@0"
I20260810 03:44:08.603272 8401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "f1e6642ee8ab4469a223efb7ea2f5cd1"
format_stamp: "Formatted at 2026-08-10 03:44:08 on dist-test-slave-ss4n"
server_key: "cf4c328f2924ae7c851348ba620d9c3a"
server_key_iv: "0074872278d7242de41ff5971b76c0d7"
server_key_version: "encryptionkey@0"
I20260810 03:44:08.607187 8401 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:44:08.609738 8418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:08.611057 8401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:44:08.611179 8401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "f1e6642ee8ab4469a223efb7ea2f5cd1"
format_stamp: "Formatted at 2026-08-10 03:44:08 on dist-test-slave-ss4n"
server_key: "cf4c328f2924ae7c851348ba620d9c3a"
server_key_iv: "0074872278d7242de41ff5971b76c0d7"
server_key_version: "encryptionkey@0"
I20260810 03:44:08.611328 8401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:08.637353 8401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:08.641057 8401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:08.641288 8401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:08.641973 8401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:08.642890 8401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:08.642938 8401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:08.643013 8401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:08.643044 8401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:08.652750 8401 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:34159
I20260810 03:44:08.652776 8546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:34159 every 8 connection(s)
I20260810 03:44:08.653740 8401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:44:08.661145 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 8401
I20260810 03:44:08.661293 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:44:08.661635 22205 external_mini_cluster.cc:1428] Setting key e56618a5030e8456af3962904827b610
Aug 10 03:44:08 dist-test-slave-ss4n krb5kdc[8282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333448, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:08.669827 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:08.655822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:60469 (local address 127.21.175.126:40377)
0810 03:44:08.655975 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:08.655979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:08.657005 (+ 1026us) server_negotiation.cc:407] Connection header received
0810 03:44:08.658497 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:08.658502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:08.658574 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:08.658679 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:08.659878 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:08.660452 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:08.662110 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:08.662440 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:08.662657 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:08.665630 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:08.665633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:08.665635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:08.665659 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:08.667173 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:08.667942 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:08.667945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:08.667947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:08.668044 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:08.668522 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:08.668526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:08.668527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:08.668729 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:08.668816 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:08.669590 (+ 774us) server_negotiation.cc:299] Negotiation successful
0810 03:44:08.669701 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 03:44:08.671144 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:08.656284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:40377 (local address 127.21.175.65:60469)
0810 03:44:08.656852 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:08.656896 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:08.658278 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:08.658871 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:08.658878 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:08.659238 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:08.659727 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:08.659741 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:08.660585 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:08.660589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:08.661934 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:08.661945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:08.662678 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:08.662682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:08.662883 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:08.663505 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:08.663528 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:08.665380 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:08.667299 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:08.667309 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:08.667329 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:08.667726 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:08.668160 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:08.668167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:08.668172 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:08.668380 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:08.668849 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:08.668858 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:08.669251 (+ 393us) client_negotiation.cc:769] Sending connection context
0810 03:44:08.669522 (+ 271us) client_negotiation.cc:240] Negotiation successful
0810 03:44:08.669945 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":145,"threads_started":1}
I20260810 03:44:08.672562 8547 heartbeater.cc:347] Connected to a master server at 127.21.175.126:40377
I20260810 03:44:08.672825 8547 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
I20260810 03:44:08.673532 8547 heartbeater.cc:511] Master 127.21.175.126:40377 requested a full tablet report, sending...
I20260810 03:44:08.675325 8330 ts_manager.cc:194] Registered new tserver with Master: f1e6642ee8ab4469a223efb7ea2f5cd1 (127.21.175.65:34159)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
I20260810 03:44:08.676699 8330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:60469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:44:08.696529 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:40377
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:08.816103 8553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:08.816385 8553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:08.816429 8553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:08.820473 8553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:08.820541 8553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:08.820607 8553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:44:08.826140 8553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:40377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:08.827257 8553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:08.828282 8553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:08.838310 8558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:08.838310 8561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:08.838326 8559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:08.838748 8553 server_base.cc:1034] running on GCE node
I20260810 03:44:08.839179 8553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:08.839854 8553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:08.841056 8553 hybrid_clock.cc:648] HybridClock initialized: now 1786333448841029 us; error 45 us; skew 500 ppm
Aug 10 03:44:08 dist-test-slave-ss4n krb5kdc[8282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333448, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:08.848304 8553 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:44:08.849264 8553 webserver.cc:460] Webserver started at http://127.21.175.66:44379/ using document root <none> and password file <none>
I20260810 03:44:08.849913 8553 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:08.850008 8553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:08.850247 8553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:08.852320 8553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "64fb9512bcad4eaeb81e2e8a09704840"
format_stamp: "Formatted at 2026-08-10 03:44:08 on dist-test-slave-ss4n"
server_key: "ca467f37d5eb9c8c24de59e65043519a"
server_key_iv: "2f85272bc47155a53fa2bb9e658556d5"
server_key_version: "encryptionkey@0"
I20260810 03:44:08.852906 8553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "64fb9512bcad4eaeb81e2e8a09704840"
format_stamp: "Formatted at 2026-08-10 03:44:08 on dist-test-slave-ss4n"
server_key: "ca467f37d5eb9c8c24de59e65043519a"
server_key_iv: "2f85272bc47155a53fa2bb9e658556d5"
server_key_version: "encryptionkey@0"
I20260810 03:44:08.856869 8553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:44:08.859781 8570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:08.861169 8553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:44:08.861337 8553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "64fb9512bcad4eaeb81e2e8a09704840"
format_stamp: "Formatted at 2026-08-10 03:44:08 on dist-test-slave-ss4n"
server_key: "ca467f37d5eb9c8c24de59e65043519a"
server_key_iv: "2f85272bc47155a53fa2bb9e658556d5"
server_key_version: "encryptionkey@0"
I20260810 03:44:08.861475 8553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:08.875386 8553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:08.878551 8553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:08.878759 8553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:08.879359 8553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:08.880214 8553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:08.880285 8553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:08.880358 8553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:08.880411 8553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:08.889853 8553 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:32929
I20260810 03:44:08.889889 8698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:32929 every 8 connection(s)
I20260810 03:44:08.890828 8553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:44:08.894996 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 8553
I20260810 03:44:08.895170 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:44:08.895566 22205 external_mini_cluster.cc:1428] Setting key e06c551dffc1b6a60ef473cc7a697bb0
Aug 10 03:44:08 dist-test-slave-ss4n krb5kdc[8282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333448, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
I20260810 03:44:08.909322 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:08.893098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:39499 (local address 127.21.175.126:40377)
0810 03:44:08.893263 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:08.893269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:08.894412 (+ 1143us) server_negotiation.cc:407] Connection header received
0810 03:44:08.896598 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:08.896602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:08.896666 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:08.896770 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:08.898705 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:08.899362 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:08.900804 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:08.901157 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:08.901266 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:08.904486 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:08.904491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:08.904496 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:08.904541 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:08.906635 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:08.907298 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:08.907302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:08.907304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:08.907421 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:08.907889 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:08.907896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:08.907902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:08.908214 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:08.908323 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:08.909036 (+ 713us) server_negotiation.cc:299] Negotiation successful
0810 03:44:08.909177 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 03:44:08.910331 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:08.893655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:40377 (local address 127.21.175.66:39499)
0810 03:44:08.894225 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:08.894274 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:08.896284 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:08.896997 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:08.897006 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:08.897684 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:08.898491 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:08.898528 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:08.899674 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:08.899681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:08.900664 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:08.900675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:08.901335 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:08.901338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:08.901647 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:08.902335 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:08.902352 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:08.904257 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:08.906802 (+ 2545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:08.906809 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:08.906826 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:08.907153 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:08.907585 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:08.907587 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:08.907589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:08.907753 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:08.908337 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:08.908342 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:08.908757 (+ 415us) client_negotiation.cc:769] Sending connection context
0810 03:44:08.908932 (+ 175us) client_negotiation.cc:240] Negotiation successful
0810 03:44:08.909244 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":141,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
I20260810 03:44:08.911921 8699 heartbeater.cc:347] Connected to a master server at 127.21.175.126:40377
I20260810 03:44:08.912338 8699 heartbeater.cc:464] Registering TS with master...
I20260810 03:44:08.913076 8699 heartbeater.cc:511] Master 127.21.175.126:40377 requested a full tablet report, sending...
I20260810 03:44:08.914528 8330 ts_manager.cc:194] Registered new tserver with Master: 64fb9512bcad4eaeb81e2e8a09704840 (127.21.175.66:32929)
I20260810 03:44:08.915485 8330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:39499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:44:08.930299 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:40377
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:09.046540 8705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:09.046828 8705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:09.046871 8705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:09.051051 8705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:09.051120 8705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:09.051179 8705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:44:09.056542 8705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:40377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:09.057788 8705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:09.058790 8705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:09.069170 8714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:09.069219 8711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:09.069247 8705 server_base.cc:1034] running on GCE node
W20260810 03:44:09.069234 8712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:09.070202 8705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:09.070883 8705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:09.072098 8705 hybrid_clock.cc:648] HybridClock initialized: now 1786333449072057 us; error 48 us; skew 500 ppm
Aug 10 03:44:09 dist-test-slave-ss4n krb5kdc[8282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333449, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:09.075704 8705 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:44:09.076767 8705 webserver.cc:460] Webserver started at http://127.21.175.67:46619/ using document root <none> and password file <none>
I20260810 03:44:09.077428 8705 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:09.077572 8705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:09.077853 8705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:09.080183 8705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "f495b762c7d540c69731283fc6dfeb28"
format_stamp: "Formatted at 2026-08-10 03:44:09 on dist-test-slave-ss4n"
server_key: "3414c33bb5d5d3f885727be07e3021d9"
server_key_iv: "7cfef32e65038abb9c138c1ac558410e"
server_key_version: "encryptionkey@0"
I20260810 03:44:09.080849 8705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "f495b762c7d540c69731283fc6dfeb28"
format_stamp: "Formatted at 2026-08-10 03:44:09 on dist-test-slave-ss4n"
server_key: "3414c33bb5d5d3f885727be07e3021d9"
server_key_iv: "7cfef32e65038abb9c138c1ac558410e"
server_key_version: "encryptionkey@0"
I20260810 03:44:09.085137 8705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260810 03:44:09.088045 8722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:09.089368 8705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 03:44:09.089469 8705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "f495b762c7d540c69731283fc6dfeb28"
format_stamp: "Formatted at 2026-08-10 03:44:09 on dist-test-slave-ss4n"
server_key: "3414c33bb5d5d3f885727be07e3021d9"
server_key_iv: "7cfef32e65038abb9c138c1ac558410e"
server_key_version: "encryptionkey@0"
I20260810 03:44:09.089634 8705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:09.107218 8705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:09.110811 8705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:09.110970 8705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:09.111509 8705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:09.112293 8705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:09.112341 8705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:09.112380 8705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:09.112396 8705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:09.121622 8705 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:37649
I20260810 03:44:09.121709 8848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:37649 every 8 connection(s)
I20260810 03:44:09.122620 8705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:44:09.128664 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 8705
I20260810 03:44:09.128846 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:44:09.129179 22205 external_mini_cluster.cc:1428] Setting key 1e3ee9119ffff9d2af5851ca541a0bf3
Aug 10 03:44:09 dist-test-slave-ss4n krb5kdc[8282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333449, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:09.138803 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:09.124601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:48157 (local address 127.21.175.126:40377)
0810 03:44:09.124751 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:09.124754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:09.125697 (+ 943us) server_negotiation.cc:407] Connection header received
0810 03:44:09.127165 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:09.127168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:09.127221 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:09.127300 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:09.128843 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.129544 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.131223 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.131647 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.131765 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:09.134535 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:09.134539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:09.134541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:09.134572 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:09.136491 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:09.137057 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:09.137060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:09.137061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:09.137162 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:09.137628 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:09.137631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:09.137633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:09.137853 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:09.137971 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:09.138538 (+ 567us) server_negotiation.cc:299] Negotiation successful
0810 03:44:09.138665 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:44:09.139628 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:09.124987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:40377 (local address 127.21.175.67:48157)
0810 03:44:09.125463 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:09.125570 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:09.126898 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:09.127509 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:09.127516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:09.127944 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:09.128636 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:09.128661 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.129704 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.129709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:09.131060 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:09.131071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.131829 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.131833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:09.131983 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:09.132639 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:09.132658 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:09.134282 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:09.136652 (+ 2370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:09.136657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:09.136671 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:09.136939 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:09.137288 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:09.137290 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:09.137292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:09.137445 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:09.137987 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:09.137992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:09.138253 (+ 261us) client_negotiation.cc:769] Sending connection context
0810 03:44:09.138411 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 03:44:09.138732 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
I20260810 03:44:09.140897 8849 heartbeater.cc:347] Connected to a master server at 127.21.175.126:40377
I20260810 03:44:09.141158 8849 heartbeater.cc:464] Registering TS with master...
I20260810 03:44:09.141739 8849 heartbeater.cc:511] Master 127.21.175.126:40377 requested a full tablet report, sending...
I20260810 03:44:09.142961 8330 ts_manager.cc:194] Registered new tserver with Master: f495b762c7d540c69731283fc6dfeb28 (127.21.175.67:37649)
I20260810 03:44:09.143625 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:44:09.143774 8330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:48157
I20260810 03:44:09.153402 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:09.145935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52366 (local address 127.21.175.126:40377)
0810 03:44:09.146078 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:09.146083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:09.146296 (+ 213us) server_negotiation.cc:407] Connection header received
0810 03:44:09.146387 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:09.146390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:09.146427 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:09.146485 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:09.147168 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.147803 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.148674 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.148892 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.148973 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:09.150427 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:09.150431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:09.150433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:09.150453 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:09.151751 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:09.152139 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:09.152141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:09.152143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:09.152213 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:09.152436 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:09.152438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:09.152439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:09.152688 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:09.152796 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:09.153039 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 03:44:09.153190 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:44:09.156627 8330 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:44:09.159745 8330 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:44:09.174697 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:09.168677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:32929 (local address 127.0.0.1:33878)
0810 03:44:09.169095 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:09.169145 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:09.169270 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:09.170255 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:09.170261 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:09.170287 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:09.170423 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:09.170432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.171685 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.171691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:09.172948 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:09.172960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.174123 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.174127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:09.174250 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:09.174296 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 03:44:09.174415 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 03:44:09.174540 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":67,"threads_started":1}
I20260810 03:44:09.174741 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:09.168765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33878 (local address 127.21.175.66:32929)
0810 03:44:09.169485 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:09.169515 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:09.169541 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:44:09.169626 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:09.169631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:09.169850 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:09.170012 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:09.170563 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.171563 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.173094 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.174012 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.174097 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:09.174211 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:09.174480 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 03:44:09.174623 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":169,"threads_started":1}
I20260810 03:44:09.175581 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:09.169597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:37649 (local address 127.0.0.1:53452)
0810 03:44:09.170334 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:09.170348 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:09.170435 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:09.170840 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:09.170842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:09.170860 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:09.170976 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:09.170982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.172631 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.172635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:09.173797 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:09.173809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.175154 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.175159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:09.175257 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:09.175277 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 03:44:09.175371 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 03:44:09.175443 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":18,"thread_start_us":104,"threads_started":1}
I20260810 03:44:09.175626 8867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:09.169790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53452 (local address 127.21.175.67:37649)
0810 03:44:09.170239 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:09.170251 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:09.170418 (+ 167us) server_negotiation.cc:407] Connection header received
0810 03:44:09.170551 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:09.170562 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:09.170710 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:09.170861 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:09.171107 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.172269 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.173947 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.174824 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.175001 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:09.175153 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:09.175408 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 03:44:09.175498 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":135,"threads_started":1}
I20260810 03:44:09.176870 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:09.168720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:34159 (local address 127.0.0.1:35858)
0810 03:44:09.170045 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:09.170059 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:09.170153 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:09.171010 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:09.171013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:09.171030 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:09.171121 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:09.171127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.172502 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.172505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:09.173798 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:09.173810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.176432 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.176435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:09.176531 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:09.176565 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 03:44:09.176663 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 03:44:09.176734 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1253,"mutex_wait_us":61,"thread_start_us":117,"threads_started":1}
I20260810 03:44:09.177296 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:09.169465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.21.175.65:34159)
0810 03:44:09.170373 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:09.170390 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:09.170413 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:44:09.170491 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:09.170500 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:09.170718 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:09.170870 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:09.171247 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.172375 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.175034 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.176130 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.176233 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:09.176371 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:09.177081 (+ 710us) server_negotiation.cc:299] Negotiation successful
0810 03:44:09.177188 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":110,"threads_started":1}
I20260810 03:44:09.178313 8623 tablet_service.cc:1467] Processing CreateTablet for tablet cc4cc97e327a4f37a1f10f016487cdd1 (DEFAULT_TABLE table=test-table [id=f8382d42536b41cd89c2a5227b0689e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:09.179495 8623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc4cc97e327a4f37a1f10f016487cdd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:09.179279 8775 tablet_service.cc:1467] Processing CreateTablet for tablet cc4cc97e327a4f37a1f10f016487cdd1 (DEFAULT_TABLE table=test-table [id=f8382d42536b41cd89c2a5227b0689e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:09.180219 8775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc4cc97e327a4f37a1f10f016487cdd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:09.179909 8471 tablet_service.cc:1467] Processing CreateTablet for tablet cc4cc97e327a4f37a1f10f016487cdd1 (DEFAULT_TABLE table=test-table [id=f8382d42536b41cd89c2a5227b0689e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:09.180806 8471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc4cc97e327a4f37a1f10f016487cdd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:09.186256 8868 tablet_bootstrap.cc:492] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28: Bootstrap starting.
I20260810 03:44:09.188123 8870 tablet_bootstrap.cc:492] T cc4cc97e327a4f37a1f10f016487cdd1 P f1e6642ee8ab4469a223efb7ea2f5cd1: Bootstrap starting.
I20260810 03:44:09.188426 8868 tablet_bootstrap.cc:654] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:09.189071 8869 tablet_bootstrap.cc:492] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840: Bootstrap starting.
I20260810 03:44:09.189738 8868 log.cc:824] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:09.190315 8870 tablet_bootstrap.cc:654] T cc4cc97e327a4f37a1f10f016487cdd1 P f1e6642ee8ab4469a223efb7ea2f5cd1: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:09.191119 8869 tablet_bootstrap.cc:654] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:09.191151 8870 log.cc:824] T cc4cc97e327a4f37a1f10f016487cdd1 P f1e6642ee8ab4469a223efb7ea2f5cd1: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:09.192109 8868 tablet_bootstrap.cc:492] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28: No bootstrap required, opened a new log
I20260810 03:44:09.192315 8868 ts_tablet_manager.cc:1397] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:44:09.192365 8869 log.cc:824] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:09.193130 8870 tablet_bootstrap.cc:492] T cc4cc97e327a4f37a1f10f016487cdd1 P f1e6642ee8ab4469a223efb7ea2f5cd1: No bootstrap required, opened a new log
I20260810 03:44:09.193339 8870 ts_tablet_manager.cc:1397] T cc4cc97e327a4f37a1f10f016487cdd1 P f1e6642ee8ab4469a223efb7ea2f5cd1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 03:44:09.194830 8869 tablet_bootstrap.cc:492] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840: No bootstrap required, opened a new log
I20260810 03:44:09.195029 8869 ts_tablet_manager.cc:1397] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:44:09.195825 8868 raft_consensus.cc:348] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fb9512bcad4eaeb81e2e8a09704840" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 32929 } } peers { permanent_uuid: "f1e6642ee8ab4469a223efb7ea2f5cd1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34159 } } peers { permanent_uuid: "f495b762c7d540c69731283fc6dfeb28" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 37649 } }
I20260810 03:44:09.196079 8868 raft_consensus.cc:374] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:09.196132 8868 raft_consensus.cc:729] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f495b762c7d540c69731283fc6dfeb28, State: Initialized, Role: FOLLOWER
I20260810 03:44:09.196585 8868 consensus_queue.cc:260] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fb9512bcad4eaeb81e2e8a09704840" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 32929 } } peers { permanent_uuid: "f1e6642ee8ab4469a223efb7ea2f5cd1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34159 } } peers { permanent_uuid: "f495b762c7d540c69731283fc6dfeb28" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 37649 } }
I20260810 03:44:09.197310 8849 heartbeater.cc:503] Master 127.21.175.126:40377 was elected leader, sending a full tablet report...
I20260810 03:44:09.197774 8868 ts_tablet_manager.cc:1428] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 03:44:09.198613 8870 raft_consensus.cc:348] T cc4cc97e327a4f37a1f10f016487cdd1 P f1e6642ee8ab4469a223efb7ea2f5cd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fb9512bcad4eaeb81e2e8a09704840" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 32929 } } peers { permanent_uuid: "f1e6642ee8ab4469a223efb7ea2f5cd1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34159 } } peers { permanent_uuid: "f495b762c7d540c69731283fc6dfeb28" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 37649 } }
I20260810 03:44:09.198925 8870 raft_consensus.cc:374] T cc4cc97e327a4f37a1f10f016487cdd1 P f1e6642ee8ab4469a223efb7ea2f5cd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:09.198982 8870 raft_consensus.cc:729] T cc4cc97e327a4f37a1f10f016487cdd1 P f1e6642ee8ab4469a223efb7ea2f5cd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e6642ee8ab4469a223efb7ea2f5cd1, State: Initialized, Role: FOLLOWER
I20260810 03:44:09.198818 8869 raft_consensus.cc:348] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fb9512bcad4eaeb81e2e8a09704840" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 32929 } } peers { permanent_uuid: "f1e6642ee8ab4469a223efb7ea2f5cd1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34159 } } peers { permanent_uuid: "f495b762c7d540c69731283fc6dfeb28" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 37649 } }
I20260810 03:44:09.199074 8869 raft_consensus.cc:374] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:09.199172 8869 raft_consensus.cc:729] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fb9512bcad4eaeb81e2e8a09704840, State: Initialized, Role: FOLLOWER
I20260810 03:44:09.199431 8870 consensus_queue.cc:260] T cc4cc97e327a4f37a1f10f016487cdd1 P f1e6642ee8ab4469a223efb7ea2f5cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fb9512bcad4eaeb81e2e8a09704840" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 32929 } } peers { permanent_uuid: "f1e6642ee8ab4469a223efb7ea2f5cd1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34159 } } peers { permanent_uuid: "f495b762c7d540c69731283fc6dfeb28" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 37649 } }
I20260810 03:44:09.199645 8869 consensus_queue.cc:260] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fb9512bcad4eaeb81e2e8a09704840" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 32929 } } peers { permanent_uuid: "f1e6642ee8ab4469a223efb7ea2f5cd1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34159 } } peers { permanent_uuid: "f495b762c7d540c69731283fc6dfeb28" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 37649 } }
I20260810 03:44:09.200201 8547 heartbeater.cc:503] Master 127.21.175.126:40377 was elected leader, sending a full tablet report...
I20260810 03:44:09.200587 8870 ts_tablet_manager.cc:1428] T cc4cc97e327a4f37a1f10f016487cdd1 P f1e6642ee8ab4469a223efb7ea2f5cd1: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 03:44:09.200599 8869 ts_tablet_manager.cc:1428] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:44:09.200898 8699 heartbeater.cc:503] Master 127.21.175.126:40377 was elected leader, sending a full tablet report...
W20260810 03:44:09.373677 8851 tablet.cc:2369] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:44:09.392164 8700 tablet.cc:2369] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:44:09.405216 8548 tablet.cc:2369] T cc4cc97e327a4f37a1f10f016487cdd1 P f1e6642ee8ab4469a223efb7ea2f5cd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:44:09.432662 8877 raft_consensus.cc:482] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:44:09.432885 8877 raft_consensus.cc:504] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fb9512bcad4eaeb81e2e8a09704840" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 32929 } } peers { permanent_uuid: "f1e6642ee8ab4469a223efb7ea2f5cd1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34159 } } peers { permanent_uuid: "f495b762c7d540c69731283fc6dfeb28" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 37649 } }
I20260810 03:44:09.434128 8877 leader_election.cc:288] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e6642ee8ab4469a223efb7ea2f5cd1 (127.21.175.65:34159), f495b762c7d540c69731283fc6dfeb28 (127.21.175.67:37649)
I20260810 03:44:09.438297 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:09.434486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:50231 (local address 127.21.175.65:34159)
0810 03:44:09.434645 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:09.434650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:09.434850 (+ 200us) server_negotiation.cc:407] Connection header received
0810 03:44:09.435064 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:09.435068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:09.435128 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:09.435172 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:09.435554 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.436059 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.437227 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.437723 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.437856 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:09.437912 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:09.438096 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 03:44:09.438170 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:44:09.438318 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:09.434374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:34159 (local address 127.21.175.66:50231)
0810 03:44:09.434746 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:09.434758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:09.434905 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:09.435294 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:09.435296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:09.435310 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:09.435423 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:09.435427 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.436187 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.436190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:09.437088 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:09.437098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.437851 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.437854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:09.437959 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:09.437979 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 03:44:09.438080 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 03:44:09.438157 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":54,"spinlock_wait_cycles":9600,"thread_start_us":123,"threads_started":1}
I20260810 03:44:09.438431 8867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:09.434611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:35985 (local address 127.21.175.67:37649)
0810 03:44:09.434732 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:09.434737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:09.434976 (+ 239us) server_negotiation.cc:407] Connection header received
0810 03:44:09.435126 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:09.435132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:09.435189 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:09.435283 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:09.435641 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.436182 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.437330 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.437807 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.437961 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:09.438018 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:09.438203 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 03:44:09.438276 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 03:44:09.438702 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:09.434397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:37649 (local address 127.21.175.66:35985)
0810 03:44:09.434824 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:09.434837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:09.434989 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:09.435296 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:09.435299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:09.435347 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:09.435517 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:09.435523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.436306 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.436311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:09.437189 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:09.437198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.437942 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.437946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:09.438056 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:09.438082 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:44:09.438186 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 03:44:09.438264 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"spinlock_wait_cycles":3200,"thread_start_us":99,"threads_started":1}
I20260810 03:44:09.439239 8492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4cc97e327a4f37a1f10f016487cdd1" candidate_uuid: "64fb9512bcad4eaeb81e2e8a09704840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e6642ee8ab4469a223efb7ea2f5cd1" is_pre_election: true
I20260810 03:44:09.439314 8799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4cc97e327a4f37a1f10f016487cdd1" candidate_uuid: "64fb9512bcad4eaeb81e2e8a09704840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f495b762c7d540c69731283fc6dfeb28" is_pre_election: true
I20260810 03:44:09.439579 8492 raft_consensus.cc:2454] T cc4cc97e327a4f37a1f10f016487cdd1 P f1e6642ee8ab4469a223efb7ea2f5cd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64fb9512bcad4eaeb81e2e8a09704840 in term 0.
I20260810 03:44:09.439680 8799 raft_consensus.cc:2454] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64fb9512bcad4eaeb81e2e8a09704840 in term 0.
I20260810 03:44:09.440148 8574 leader_election.cc:302] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64fb9512bcad4eaeb81e2e8a09704840, f1e6642ee8ab4469a223efb7ea2f5cd1; no voters:
I20260810 03:44:09.440429 8877 raft_consensus.cc:2781] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:44:09.440537 8877 raft_consensus.cc:482] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:44:09.440609 8877 raft_consensus.cc:3037] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:09.441816 8877 raft_consensus.cc:504] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fb9512bcad4eaeb81e2e8a09704840" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 32929 } } peers { permanent_uuid: "f1e6642ee8ab4469a223efb7ea2f5cd1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34159 } } peers { permanent_uuid: "f495b762c7d540c69731283fc6dfeb28" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 37649 } }
I20260810 03:44:09.442246 8877 leader_election.cc:288] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [CANDIDATE]: Term 1 election: Requested vote from peers f1e6642ee8ab4469a223efb7ea2f5cd1 (127.21.175.65:34159), f495b762c7d540c69731283fc6dfeb28 (127.21.175.67:37649)
I20260810 03:44:09.442806 8799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4cc97e327a4f37a1f10f016487cdd1" candidate_uuid: "64fb9512bcad4eaeb81e2e8a09704840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f495b762c7d540c69731283fc6dfeb28"
I20260810 03:44:09.442960 8799 raft_consensus.cc:3037] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:09.444293 8799 raft_consensus.cc:2454] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fb9512bcad4eaeb81e2e8a09704840 in term 1.
I20260810 03:44:09.444656 8492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4cc97e327a4f37a1f10f016487cdd1" candidate_uuid: "64fb9512bcad4eaeb81e2e8a09704840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e6642ee8ab4469a223efb7ea2f5cd1"
I20260810 03:44:09.444742 8573 leader_election.cc:302] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64fb9512bcad4eaeb81e2e8a09704840, f495b762c7d540c69731283fc6dfeb28; no voters:
I20260810 03:44:09.444824 8492 raft_consensus.cc:3037] T cc4cc97e327a4f37a1f10f016487cdd1 P f1e6642ee8ab4469a223efb7ea2f5cd1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:09.444949 8877 raft_consensus.cc:2781] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:09.445315 8877 raft_consensus.cc:686] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [term 1 LEADER]: Becoming Leader. State: Replica: 64fb9512bcad4eaeb81e2e8a09704840, State: Running, Role: LEADER
I20260810 03:44:09.445797 8877 consensus_queue.cc:237] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fb9512bcad4eaeb81e2e8a09704840" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 32929 } } peers { permanent_uuid: "f1e6642ee8ab4469a223efb7ea2f5cd1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34159 } } peers { permanent_uuid: "f495b762c7d540c69731283fc6dfeb28" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 37649 } }
I20260810 03:44:09.446290 8492 raft_consensus.cc:2454] T cc4cc97e327a4f37a1f10f016487cdd1 P f1e6642ee8ab4469a223efb7ea2f5cd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fb9512bcad4eaeb81e2e8a09704840 in term 1.
I20260810 03:44:09.449357 8329 catalog_manager.cc:5495] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64fb9512bcad4eaeb81e2e8a09704840 (127.21.175.66). New cstate: current_term: 1 leader_uuid: "64fb9512bcad4eaeb81e2e8a09704840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fb9512bcad4eaeb81e2e8a09704840" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 32929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e6642ee8ab4469a223efb7ea2f5cd1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f495b762c7d540c69731283fc6dfeb28" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 37649 } health_report { overall_health: UNKNOWN } } }
I20260810 03:44:09.469897 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:09.466126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33886 (local address 127.21.175.66:32929)
0810 03:44:09.466277 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:09.466281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:09.466296 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:44:09.466418 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:09.466421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:09.466481 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:09.466625 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:44:09.466947 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.467454 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.468374 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.468589 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.468713 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:09.469014 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:44:09.469464 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:44:09.469694 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:09.469726 (+ 32us) server_negotiation.cc:299] Negotiation successful
0810 03:44:09.469782 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 03:44:09.476413 8492 raft_consensus.cc:1261] T cc4cc97e327a4f37a1f10f016487cdd1 P f1e6642ee8ab4469a223efb7ea2f5cd1 [term 1 FOLLOWER]: Refusing update from remote peer 64fb9512bcad4eaeb81e2e8a09704840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:44:09.476408 8799 raft_consensus.cc:1261] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28 [term 1 FOLLOWER]: Refusing update from remote peer 64fb9512bcad4eaeb81e2e8a09704840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:44:09.477169 8877 consensus_queue.cc:1035] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e6642ee8ab4469a223efb7ea2f5cd1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 34159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:44:09.477394 8880 consensus_queue.cc:1035] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f495b762c7d540c69731283fc6dfeb28" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 37649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:44:09.484714 8883 mvcc.cc:205] Tried to move back new op lower bound from 7316821809047388160 to 7316821808933277696. Current Snapshot: MvccSnapshot[applied={T|T < 7316821809047388160}]
I20260810 03:44:09.493969 8329 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:44:09.494182 8329 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:44:09.496634 8329 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c4ac41eca45464b9713ac97eecd7b22: Sending DeleteTablet for 3 replicas of tablet cc4cc97e327a4f37a1f10f016487cdd1
I20260810 03:44:09.497608 8471 tablet_service.cc:1514] Processing DeleteTablet for tablet cc4cc97e327a4f37a1f10f016487cdd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:09 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:35858
I20260810 03:44:09.497676 8623 tablet_service.cc:1514] Processing DeleteTablet for tablet cc4cc97e327a4f37a1f10f016487cdd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:09 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:33878
I20260810 03:44:09.497710 8775 tablet_service.cc:1514] Processing DeleteTablet for tablet cc4cc97e327a4f37a1f10f016487cdd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:09 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:53452
I20260810 03:44:09.498227 8894 tablet_replica.cc:330] T cc4cc97e327a4f37a1f10f016487cdd1 P f1e6642ee8ab4469a223efb7ea2f5cd1: stopping tablet replica
I20260810 03:44:09.498368 8896 tablet_replica.cc:330] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28: stopping tablet replica
I20260810 03:44:09.498612 8893 tablet_replica.cc:330] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840: stopping tablet replica
I20260810 03:44:09.498713 8896 raft_consensus.cc:2229] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:44:09.499012 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 8401
I20260810 03:44:09.499065 8896 raft_consensus.cc:2258] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:44:09.499379 8893 raft_consensus.cc:2229] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:44:09.499962 8893 raft_consensus.cc:2258] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:44:09.502388 8893 ts_tablet_manager.cc:1905] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:44:09.502892 8896 ts_tablet_manager.cc:1905] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:44:09.505659 8896 ts_tablet_manager.cc:1918] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:44:09.505771 8896 log.cc:1196] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/cc4cc97e327a4f37a1f10f016487cdd1
I20260810 03:44:09.506127 8896 ts_tablet_manager.cc:1939] T cc4cc97e327a4f37a1f10f016487cdd1 P f495b762c7d540c69731283fc6dfeb28: Deleting consensus metadata
I20260810 03:44:09.506909 8893 ts_tablet_manager.cc:1918] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:44:09.507099 8893 log.cc:1196] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/cc4cc97e327a4f37a1f10f016487cdd1
I20260810 03:44:09.507506 8893 ts_tablet_manager.cc:1939] T cc4cc97e327a4f37a1f10f016487cdd1 P 64fb9512bcad4eaeb81e2e8a09704840: Deleting consensus metadata
I20260810 03:44:09.507632 8315 catalog_manager.cc:4841] TS f495b762c7d540c69731283fc6dfeb28 (127.21.175.67:37649): tablet cc4cc97e327a4f37a1f10f016487cdd1 (table test-table [id=f8382d42536b41cd89c2a5227b0689e0]) successfully deleted
I20260810 03:44:09.509013 8313 catalog_manager.cc:4841] TS 64fb9512bcad4eaeb81e2e8a09704840 (127.21.175.66:32929): tablet cc4cc97e327a4f37a1f10f016487cdd1 (table test-table [id=f8382d42536b41cd89c2a5227b0689e0]) successfully deleted
W20260810 03:44:09.512077 8316 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.175.65:34159 (error 108)
I20260810 03:44:09.513037 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 8553
I20260810 03:44:09.513336 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:09.512985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:34159 (local address 127.0.0.1:35860)
0810 03:44:09.513149 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:09.513231 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:34159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 03:44:09.515199 8316 catalog_manager.cc:4584] TS f1e6642ee8ab4469a223efb7ea2f5cd1 (127.21.175.65:34159): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc4cc97e327a4f37a1f10f016487cdd1: Network error: Client connection negotiation failed: client connection to 127.21.175.65:34159: connect: Connection refused (error 111)
I20260810 03:44:09.520800 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 8705
I20260810 03:44:09.530264 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 8296
2026-08-10T03:44:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2645 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T03:44:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:09Z Disabled control of system clock
I20260810 03:44:09.566335 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:35425
--webserver_interface=127.21.175.126
--webserver_port=0
--builtin_ntp_servers=127.21.175.84:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:35425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:44:09.690837 8901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:09.691174 8901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:09.691215 8901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:09.696035 8901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:09.696125 8901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:09.696143 8901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:09.696163 8901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:09.696180 8901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:09.706243 8901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:35425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:35425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:09.707895 8901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:09.709766 8901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:09.719815 8910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:09.719815 8908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:09.719857 8907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:09.720198 8901 server_base.cc:1034] running on GCE node
I20260810 03:44:09.720773 8901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:09.722066 8901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:09.723284 8901 hybrid_clock.cc:648] HybridClock initialized: now 1786333449723269 us; error 36 us; skew 500 ppm
I20260810 03:44:09.725399 8901 webserver.cc:460] Webserver started at http://127.21.175.126:34527/ using document root <none> and password file <none>
I20260810 03:44:09.726140 8901 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:09.726277 8901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:09.726631 8901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:09.728972 8901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "5a0ed26705d548f4906df42f2d9aa6c2"
format_stamp: "Formatted at 2026-08-10 03:44:09 on dist-test-slave-ss4n"
server_key: "ce5ec10fdacc8883319c0cb574becd11"
server_key_iv: "48061da78ac0ad06d405987178f43b07"
server_key_version: "encryptionkey@0"
I20260810 03:44:09.729737 8901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "5a0ed26705d548f4906df42f2d9aa6c2"
format_stamp: "Formatted at 2026-08-10 03:44:09 on dist-test-slave-ss4n"
server_key: "ce5ec10fdacc8883319c0cb574becd11"
server_key_iv: "48061da78ac0ad06d405987178f43b07"
server_key_version: "encryptionkey@0"
I20260810 03:44:09.734091 8901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:44:09.736874 8918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:09.738416 8901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:44:09.738605 8901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "5a0ed26705d548f4906df42f2d9aa6c2"
format_stamp: "Formatted at 2026-08-10 03:44:09 on dist-test-slave-ss4n"
server_key: "ce5ec10fdacc8883319c0cb574becd11"
server_key_iv: "48061da78ac0ad06d405987178f43b07"
server_key_version: "encryptionkey@0"
I20260810 03:44:09.738740 8901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:09.774215 8901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:09.775071 8901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:09.775307 8901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:09.783332 8901 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:35425
I20260810 03:44:09.783363 8980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:35425 every 8 connection(s)
I20260810 03:44:09.784408 8901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:09.787880 8981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:09.794041 8981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2: Bootstrap starting.
I20260810 03:44:09.794055 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 8901
I20260810 03:44:09.794219 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:09.794516 22205 external_mini_cluster.cc:1428] Setting key e474eb25f0e6a2a91bb6269f5e94e73b
I20260810 03:44:09.796800 8981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:09.797768 8981 log.cc:824] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:09.800312 8981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2: No bootstrap required, opened a new log
I20260810 03:44:09.805249 8981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0ed26705d548f4906df42f2d9aa6c2" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35425 } }
I20260810 03:44:09.805572 8981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:09.805639 8981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0ed26705d548f4906df42f2d9aa6c2, State: Initialized, Role: FOLLOWER
I20260810 03:44:09.806095 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:09.796060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56792 (local address 127.21.175.126:35425)
0810 03:44:09.796750 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:09.796767 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:09.796805 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:44:09.797662 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:09.797683 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:09.798168 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:09.798522 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:09.799011 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.800575 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.801687 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:09.802532 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:09.802716 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:09.803574 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:09.803583 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:09.803603 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:09.804296 (+ 693us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:09.804377 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:09.804407 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:09.804528 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:09.804696 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 03:44:09.804994 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":148,"threads_started":1}
I20260810 03:44:09.806169 8981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0ed26705d548f4906df42f2d9aa6c2" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35425 } }
I20260810 03:44:09.806349 8981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:09.806427 8981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:09.806535 8981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:09.807754 8981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0ed26705d548f4906df42f2d9aa6c2" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35425 } }
I20260810 03:44:09.808136 8981 leader_election.cc:302] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a0ed26705d548f4906df42f2d9aa6c2; no voters:
I20260810 03:44:09.808527 8981 leader_election.cc:288] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:09.808774 8988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:09.809079 8988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [term 1 LEADER]: Becoming Leader. State: Replica: 5a0ed26705d548f4906df42f2d9aa6c2, State: Running, Role: LEADER
I20260810 03:44:09.809515 8988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0ed26705d548f4906df42f2d9aa6c2" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35425 } }
I20260810 03:44:09.810074 8981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:09.811455 8990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a0ed26705d548f4906df42f2d9aa6c2. Latest consensus state: current_term: 1 leader_uuid: "5a0ed26705d548f4906df42f2d9aa6c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0ed26705d548f4906df42f2d9aa6c2" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35425 } } }
I20260810 03:44:09.811471 8989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a0ed26705d548f4906df42f2d9aa6c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0ed26705d548f4906df42f2d9aa6c2" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 35425 } } }
I20260810 03:44:09.811622 8989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:09.811620 8990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:09.812094 8998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:09.815550 8998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:09.821556 8998 catalog_manager.cc:1347] Generated new cluster ID: dcbdc1bf4cc8401884ac74c64ef76709
I20260810 03:44:09.821632 8998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:09.829195 8998 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:09.830256 8998 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:09.842715 8998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2: Generated new TSK 0
I20260810 03:44:09.843768 8998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:09.853097 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35425
--builtin_ntp_servers=127.21.175.84:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:44:09.973599 9009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:09.973852 9009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:09.973891 9009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:09.977919 9009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:09.977978 9009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:09.978032 9009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:44:09.982633 9009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:09.983600 9009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:09.984879 9009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:09.996297 9016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:09.996312 9015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:09.996297 9018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:09.996641 9009 server_base.cc:1034] running on GCE node
I20260810 03:44:09.997165 9009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:09.997956 9009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:09.999174 9009 hybrid_clock.cc:648] HybridClock initialized: now 1786333449999153 us; error 33 us; skew 500 ppm
I20260810 03:44:10.001161 9009 webserver.cc:460] Webserver started at http://127.21.175.65:44051/ using document root <none> and password file <none>
I20260810 03:44:10.001818 9009 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:10.001910 9009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:10.002168 9009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:10.004242 9009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "49078bc534c64097b2d54d69b011683f"
format_stamp: "Formatted at 2026-08-10 03:44:09 on dist-test-slave-ss4n"
server_key: "c03f68b6a3e18fd64f52212e6fa5c93a"
server_key_iv: "1e0deb48ff095652a9a26a7757e928dc"
server_key_version: "encryptionkey@0"
I20260810 03:44:10.004830 9009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "49078bc534c64097b2d54d69b011683f"
format_stamp: "Formatted at 2026-08-10 03:44:09 on dist-test-slave-ss4n"
server_key: "c03f68b6a3e18fd64f52212e6fa5c93a"
server_key_iv: "1e0deb48ff095652a9a26a7757e928dc"
server_key_version: "encryptionkey@0"
I20260810 03:44:10.008635 9009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260810 03:44:10.011404 9024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:10.012763 9009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:44:10.012912 9009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "49078bc534c64097b2d54d69b011683f"
format_stamp: "Formatted at 2026-08-10 03:44:09 on dist-test-slave-ss4n"
server_key: "c03f68b6a3e18fd64f52212e6fa5c93a"
server_key_iv: "1e0deb48ff095652a9a26a7757e928dc"
server_key_version: "encryptionkey@0"
I20260810 03:44:10.013036 9009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:10.030228 9009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:10.030993 9009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:10.031206 9009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:10.031831 9009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:10.032814 9009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:10.032883 9009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:10.032958 9009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:10.033005 9009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:10.043144 9009 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:43225
I20260810 03:44:10.043185 9154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:43225 every 8 connection(s)
I20260810 03:44:10.044238 9009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:44:10.050783 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 9009
I20260810 03:44:10.050885 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:44:10.051163 22205 external_mini_cluster.cc:1428] Setting key ea15429c89cba5fc65780b04458fe310
I20260810 03:44:10.053387 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35425
--builtin_ntp_servers=127.21.175.84:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:44:10.054981 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.046389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:50649 (local address 127.21.175.126:35425)
0810 03:44:10.046544 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:10.046549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:10.047744 (+ 1195us) server_negotiation.cc:407] Connection header received
0810 03:44:10.049248 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:10.049252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:10.049315 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:10.049423 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:10.050417 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.051062 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.052669 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.052951 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.053091 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.054441 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:10.054444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:10.054446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:10.054460 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:10.054480 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:10.054489 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:10.054576 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:10.054758 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 03:44:10.054827 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:44:10.056116 9157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.046806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:35425 (local address 127.21.175.65:50649)
0810 03:44:10.047567 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:10.047611 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:10.048997 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:10.049642 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:10.049649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:10.049742 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:10.050263 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:10.050276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.051210 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.051214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:10.052501 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:10.052518 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.053136 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.053141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:10.053335 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.054118 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:44:10.054120 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:10.054166 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:44:10.054236 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:10.054594 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:10.054604 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:10.054611 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:44:10.054758 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 03:44:10.055130 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":129,"threads_started":1}
I20260810 03:44:10.057595 9155 heartbeater.cc:347] Connected to a master server at 127.21.175.126:35425
I20260810 03:44:10.057866 9155 heartbeater.cc:464] Registering TS with master...
I20260810 03:44:10.058437 9155 heartbeater.cc:511] Master 127.21.175.126:35425 requested a full tablet report, sending...
I20260810 03:44:10.060320 8940 ts_manager.cc:194] Registered new tserver with Master: 49078bc534c64097b2d54d69b011683f (127.21.175.65:43225)
I20260810 03:44:10.062181 8940 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.175.65:50649
W20260810 03:44:10.172269 9158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:10.172559 9158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:10.172600 9158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:10.176847 9158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:10.176918 9158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:10.176975 9158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:44:10.182054 9158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:10.183156 9158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:10.184508 9158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:10.194628 9163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:10.194690 9164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:10.194681 9166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:10.195032 9158 server_base.cc:1034] running on GCE node
I20260810 03:44:10.195443 9158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:10.196106 9158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:10.197301 9158 hybrid_clock.cc:648] HybridClock initialized: now 1786333450197284 us; error 30 us; skew 500 ppm
I20260810 03:44:10.199299 9158 webserver.cc:460] Webserver started at http://127.21.175.66:35659/ using document root <none> and password file <none>
I20260810 03:44:10.199918 9158 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:10.200016 9158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:10.200263 9158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:10.202554 9158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "230837dac3234917964f233c0db9da2a"
format_stamp: "Formatted at 2026-08-10 03:44:10 on dist-test-slave-ss4n"
server_key: "05fbec61a3cf6278ad0d45764676381f"
server_key_iv: "c670a48fa74aff07aacb88b92d1af43d"
server_key_version: "encryptionkey@0"
I20260810 03:44:10.203146 9158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "230837dac3234917964f233c0db9da2a"
format_stamp: "Formatted at 2026-08-10 03:44:10 on dist-test-slave-ss4n"
server_key: "05fbec61a3cf6278ad0d45764676381f"
server_key_iv: "c670a48fa74aff07aacb88b92d1af43d"
server_key_version: "encryptionkey@0"
I20260810 03:44:10.206955 9158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:44:10.209671 9172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:10.211215 9158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:44:10.211345 9158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "230837dac3234917964f233c0db9da2a"
format_stamp: "Formatted at 2026-08-10 03:44:10 on dist-test-slave-ss4n"
server_key: "05fbec61a3cf6278ad0d45764676381f"
server_key_iv: "c670a48fa74aff07aacb88b92d1af43d"
server_key_version: "encryptionkey@0"
I20260810 03:44:10.211525 9158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:10.228087 9158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:10.228932 9158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:10.229173 9158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:10.229912 9158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:10.230947 9158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:10.231022 9158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:10.231097 9158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:10.231154 9158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:10.240622 9158 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:39477
I20260810 03:44:10.240653 9302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:39477 every 8 connection(s)
I20260810 03:44:10.241758 9158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:44:10.251443 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.243803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:57521 (local address 127.21.175.126:35425)
0810 03:44:10.243949 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:10.243953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:10.245175 (+ 1222us) server_negotiation.cc:407] Connection header received
0810 03:44:10.246761 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:10.246764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:10.246824 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:10.246938 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:10.247862 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.248403 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.249610 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.249843 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.249940 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.251007 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:10.251010 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:10.251012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:10.251024 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:10.251065 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:10.251073 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:10.251135 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:10.251274 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 03:44:10.251326 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:44:10.251881 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 9158
I20260810 03:44:10.251973 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:44:10.252306 22205 external_mini_cluster.cc:1428] Setting key 2fd1c64b89e5485287276f5c6c5c1235
I20260810 03:44:10.252439 9305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.244287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:35425 (local address 127.21.175.66:57521)
0810 03:44:10.245013 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:10.245056 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:10.246533 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:10.247125 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:10.247132 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:10.247219 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:10.247720 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:10.247732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.248550 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.248553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:10.249434 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:10.249444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.249995 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.249998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:10.250147 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.250757 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:44:10.250759 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:10.250800 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:44:10.250865 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:10.251151 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:10.251158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:10.251163 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 03:44:10.251256 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 03:44:10.251480 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":130,"threads_started":1}
I20260810 03:44:10.253908 9303 heartbeater.cc:347] Connected to a master server at 127.21.175.126:35425
I20260810 03:44:10.254163 9303 heartbeater.cc:464] Registering TS with master...
I20260810 03:44:10.254526 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35425
--builtin_ntp_servers=127.21.175.84:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:44:10.254756 9303 heartbeater.cc:511] Master 127.21.175.126:35425 requested a full tablet report, sending...
I20260810 03:44:10.255877 8940 ts_manager.cc:194] Registered new tserver with Master: 230837dac3234917964f233c0db9da2a (127.21.175.66:39477)
I20260810 03:44:10.256752 8940 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.175.66:57521
W20260810 03:44:10.372949 9306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:10.373245 9306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:10.373283 9306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:10.377375 9306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:10.377446 9306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:10.377519 9306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:44:10.382381 9306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:35425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:10.383421 9306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:10.384739 9306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:10.394562 9311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:10.394598 9312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:10.394598 9315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:10.394850 9306 server_base.cc:1034] running on GCE node
I20260810 03:44:10.395377 9306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:10.396011 9306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:10.397235 9306 hybrid_clock.cc:648] HybridClock initialized: now 1786333450397198 us; error 45 us; skew 500 ppm
I20260810 03:44:10.399286 9306 webserver.cc:460] Webserver started at http://127.21.175.67:46291/ using document root <none> and password file <none>
I20260810 03:44:10.399920 9306 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:10.399992 9306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:10.400622 9306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:10.403082 9306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "f4ebe2e078d74794b44d139eda89a14f"
format_stamp: "Formatted at 2026-08-10 03:44:10 on dist-test-slave-ss4n"
server_key: "0a9a428e451ded739647a7eff902ea9a"
server_key_iv: "c2defb80b914e2d6fe12b73eb5bf61cf"
server_key_version: "encryptionkey@0"
I20260810 03:44:10.403712 9306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "f4ebe2e078d74794b44d139eda89a14f"
format_stamp: "Formatted at 2026-08-10 03:44:10 on dist-test-slave-ss4n"
server_key: "0a9a428e451ded739647a7eff902ea9a"
server_key_iv: "c2defb80b914e2d6fe12b73eb5bf61cf"
server_key_version: "encryptionkey@0"
I20260810 03:44:10.407964 9306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:44:10.411026 9321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:10.412532 9306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:44:10.412716 9306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "f4ebe2e078d74794b44d139eda89a14f"
format_stamp: "Formatted at 2026-08-10 03:44:10 on dist-test-slave-ss4n"
server_key: "0a9a428e451ded739647a7eff902ea9a"
server_key_iv: "c2defb80b914e2d6fe12b73eb5bf61cf"
server_key_version: "encryptionkey@0"
I20260810 03:44:10.412840 9306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:10.438619 9306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:10.439392 9306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:10.439613 9306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:10.440251 9306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:10.441243 9306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:10.441315 9306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:10.441391 9306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:10.441433 9306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:10.450709 9306 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:46171
I20260810 03:44:10.450742 9448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:46171 every 8 connection(s)
I20260810 03:44:10.451634 9306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:44:10.452672 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 9306
I20260810 03:44:10.452783 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:44:10.453109 22205 external_mini_cluster.cc:1428] Setting key 20b068a46f37c759bc6d8dc5d328c0b0
I20260810 03:44:10.461889 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.453758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:59133 (local address 127.21.175.126:35425)
0810 03:44:10.453905 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:10.453909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:10.455390 (+ 1481us) server_negotiation.cc:407] Connection header received
0810 03:44:10.457047 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:10.457050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:10.457097 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:10.457167 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:10.458120 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.458653 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.459858 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.460083 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.460164 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.461335 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:10.461337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:10.461339 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:10.461352 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:10.461371 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:10.461379 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:10.461438 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:10.461718 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 03:44:10.461774 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 03:44:10.462800 9451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.454338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:35425 (local address 127.21.175.67:59133)
0810 03:44:10.455208 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:10.455270 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:10.456834 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:10.457316 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:10.457323 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:10.457414 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:10.457978 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:10.457992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.458766 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.458769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:10.459748 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:10.459757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.460246 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.460250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:10.460419 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.461061 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:44:10.461063 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:10.461108 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:44:10.461184 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:10.461466 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:10.461473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:10.461478 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 03:44:10.461698 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 03:44:10.461957 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":130,"threads_started":1}
I20260810 03:44:10.464076 9449 heartbeater.cc:347] Connected to a master server at 127.21.175.126:35425
I20260810 03:44:10.464303 9449 heartbeater.cc:464] Registering TS with master...
I20260810 03:44:10.464884 9449 heartbeater.cc:511] Master 127.21.175.126:35425 requested a full tablet report, sending...
I20260810 03:44:10.466092 8940 ts_manager.cc:194] Registered new tserver with Master: f4ebe2e078d74794b44d139eda89a14f (127.21.175.67:46171)
I20260810 03:44:10.466892 8940 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.175.67:59133
I20260810 03:44:10.467941 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:44:10.473941 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.469996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56806 (local address 127.21.175.126:35425)
0810 03:44:10.470160 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:10.470165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:10.470357 (+ 192us) server_negotiation.cc:407] Connection header received
0810 03:44:10.470506 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:10.470509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:10.470563 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:10.470652 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:10.470944 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.471453 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.472321 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.472542 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.472656 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.473516 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:10.473520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:10.473523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:10.473536 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:10.473562 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:10.473571 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:10.473651 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:10.473744 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 03:44:10.473799 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:44:10.476913 8940 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:44:10.479923 8940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:44:10.495021 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.488640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:46171 (local address 127.0.0.1:50600)
0810 03:44:10.489190 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:10.489243 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:10.489393 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:10.490399 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:10.490407 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:10.490440 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:10.490598 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:10.490609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.491941 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.491948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:10.493312 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:10.493324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.494480 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.494483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:10.494591 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.494627 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 03:44:10.494734 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 03:44:10.494853 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":101,"threads_started":1}
I20260810 03:44:10.495028 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.488835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50600 (local address 127.21.175.67:46171)
0810 03:44:10.489768 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:10.489790 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:10.489818 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:44:10.489903 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:10.489909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:10.490152 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:10.490385 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:10.490759 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.491803 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.493529 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.494332 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.494476 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.494704 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:10.494760 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 03:44:10.494852 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":165,"threads_started":1}
I20260810 03:44:10.496811 9462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.489447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:39477 (local address 127.0.0.1:35990)
0810 03:44:10.490665 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:10.490680 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:10.490786 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:10.491372 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:10.491375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:10.491396 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:10.491512 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:10.491517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.493216 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.493220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:10.494502 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:10.494514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.496409 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.496412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:10.496515 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.496541 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:44:10.496609 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 03:44:10.496676 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1144,"thread_start_us":136,"threads_started":1}
I20260810 03:44:10.497222 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.489820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35990 (local address 127.21.175.66:39477)
0810 03:44:10.490235 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:10.490253 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:10.490898 (+ 645us) server_negotiation.cc:407] Connection header received
0810 03:44:10.490997 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:10.491003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:10.491209 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:10.491372 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:10.491669 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.493063 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.494663 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.496287 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.496413 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.496631 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:10.496695 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 03:44:10.496802 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":149,"threads_started":1}
I20260810 03:44:10.497895 9464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.488834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:43225 (local address 127.0.0.1:34044)
0810 03:44:10.489780 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:10.489799 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:10.489904 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:10.490861 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:10.490867 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:10.490888 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:10.490992 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:10.490998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.492878 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.492882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:10.494192 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:10.494204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.497526 (+ 3322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.497530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:10.497621 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.497645 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:44:10.497715 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 03:44:10.497777 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":122,"threads_started":1}
I20260810 03:44:10.498178 9465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.489653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34044 (local address 127.21.175.65:43225)
0810 03:44:10.490361 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:10.490374 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:10.490396 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:44:10.490474 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:10.490479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:10.490692 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:10.491164 (+ 472us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:10.491328 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.492738 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.494339 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.497311 (+ 2972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.497531 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.497725 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:10.497778 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 03:44:10.497869 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":135,"threads_started":1}
I20260810 03:44:10.498638 9373 tablet_service.cc:1467] Processing CreateTablet for tablet e5a78039367a475ebb682888672b982f (DEFAULT_TABLE table=test-table [id=6eba6aad2a9a41fd8ae977e4291d2d81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:10.499848 9373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a78039367a475ebb682888672b982f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:10.500197 9225 tablet_service.cc:1467] Processing CreateTablet for tablet e5a78039367a475ebb682888672b982f (DEFAULT_TABLE table=test-table [id=6eba6aad2a9a41fd8ae977e4291d2d81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:10.501013 9078 tablet_service.cc:1467] Processing CreateTablet for tablet e5a78039367a475ebb682888672b982f (DEFAULT_TABLE table=test-table [id=6eba6aad2a9a41fd8ae977e4291d2d81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:10.501503 9225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a78039367a475ebb682888672b982f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:10.502089 9078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a78039367a475ebb682888672b982f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:10.507982 9467 tablet_bootstrap.cc:492] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a: Bootstrap starting.
I20260810 03:44:10.509147 9469 tablet_bootstrap.cc:492] T e5a78039367a475ebb682888672b982f P 49078bc534c64097b2d54d69b011683f: Bootstrap starting.
I20260810 03:44:10.509692 9468 tablet_bootstrap.cc:492] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f: Bootstrap starting.
I20260810 03:44:10.511050 9467 tablet_bootstrap.cc:654] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:10.511289 9469 tablet_bootstrap.cc:654] T e5a78039367a475ebb682888672b982f P 49078bc534c64097b2d54d69b011683f: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:10.512148 9468 tablet_bootstrap.cc:654] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:10.512220 9469 log.cc:824] T e5a78039367a475ebb682888672b982f P 49078bc534c64097b2d54d69b011683f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:10.512753 9467 log.cc:824] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:10.512992 9468 log.cc:824] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:10.514321 9469 tablet_bootstrap.cc:492] T e5a78039367a475ebb682888672b982f P 49078bc534c64097b2d54d69b011683f: No bootstrap required, opened a new log
I20260810 03:44:10.514544 9469 ts_tablet_manager.cc:1397] T e5a78039367a475ebb682888672b982f P 49078bc534c64097b2d54d69b011683f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:44:10.515060 9468 tablet_bootstrap.cc:492] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f: No bootstrap required, opened a new log
I20260810 03:44:10.515125 9467 tablet_bootstrap.cc:492] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a: No bootstrap required, opened a new log
I20260810 03:44:10.515254 9468 ts_tablet_manager.cc:1397] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:44:10.515282 9467 ts_tablet_manager.cc:1397] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 03:44:10.518875 9469 raft_consensus.cc:348] T e5a78039367a475ebb682888672b982f P 49078bc534c64097b2d54d69b011683f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4ebe2e078d74794b44d139eda89a14f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46171 } } peers { permanent_uuid: "49078bc534c64097b2d54d69b011683f" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43225 } } peers { permanent_uuid: "230837dac3234917964f233c0db9da2a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39477 } }
I20260810 03:44:10.519181 9469 raft_consensus.cc:374] T e5a78039367a475ebb682888672b982f P 49078bc534c64097b2d54d69b011683f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:10.519249 9469 raft_consensus.cc:729] T e5a78039367a475ebb682888672b982f P 49078bc534c64097b2d54d69b011683f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49078bc534c64097b2d54d69b011683f, State: Initialized, Role: FOLLOWER
I20260810 03:44:10.519796 9469 consensus_queue.cc:260] T e5a78039367a475ebb682888672b982f P 49078bc534c64097b2d54d69b011683f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4ebe2e078d74794b44d139eda89a14f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46171 } } peers { permanent_uuid: "49078bc534c64097b2d54d69b011683f" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43225 } } peers { permanent_uuid: "230837dac3234917964f233c0db9da2a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39477 } }
I20260810 03:44:10.520009 9467 raft_consensus.cc:348] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4ebe2e078d74794b44d139eda89a14f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46171 } } peers { permanent_uuid: "49078bc534c64097b2d54d69b011683f" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43225 } } peers { permanent_uuid: "230837dac3234917964f233c0db9da2a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39477 } }
I20260810 03:44:10.520284 9467 raft_consensus.cc:374] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:10.520354 9467 raft_consensus.cc:729] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230837dac3234917964f233c0db9da2a, State: Initialized, Role: FOLLOWER
I20260810 03:44:10.520670 9155 heartbeater.cc:503] Master 127.21.175.126:35425 was elected leader, sending a full tablet report...
I20260810 03:44:10.520817 9467 consensus_queue.cc:260] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4ebe2e078d74794b44d139eda89a14f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46171 } } peers { permanent_uuid: "49078bc534c64097b2d54d69b011683f" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43225 } } peers { permanent_uuid: "230837dac3234917964f233c0db9da2a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39477 } }
I20260810 03:44:10.520955 9469 ts_tablet_manager.cc:1428] T e5a78039367a475ebb682888672b982f P 49078bc534c64097b2d54d69b011683f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 03:44:10.522083 9303 heartbeater.cc:503] Master 127.21.175.126:35425 was elected leader, sending a full tablet report...
I20260810 03:44:10.522315 9468 raft_consensus.cc:348] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4ebe2e078d74794b44d139eda89a14f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46171 } } peers { permanent_uuid: "49078bc534c64097b2d54d69b011683f" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43225 } } peers { permanent_uuid: "230837dac3234917964f233c0db9da2a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39477 } }
I20260810 03:44:10.522552 9468 raft_consensus.cc:374] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:10.522612 9468 raft_consensus.cc:729] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4ebe2e078d74794b44d139eda89a14f, State: Initialized, Role: FOLLOWER
I20260810 03:44:10.521907 9467 ts_tablet_manager.cc:1428] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 03:44:10.523046 9468 consensus_queue.cc:260] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4ebe2e078d74794b44d139eda89a14f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46171 } } peers { permanent_uuid: "49078bc534c64097b2d54d69b011683f" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43225 } } peers { permanent_uuid: "230837dac3234917964f233c0db9da2a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39477 } }
I20260810 03:44:10.524070 9449 heartbeater.cc:503] Master 127.21.175.126:35425 was elected leader, sending a full tablet report...
I20260810 03:44:10.524282 9468 ts_tablet_manager.cc:1428] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
W20260810 03:44:10.545585 9156 tablet.cc:2369] T e5a78039367a475ebb682888672b982f P 49078bc534c64097b2d54d69b011683f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:44:10.594373 9475 raft_consensus.cc:482] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:44:10.594573 9475 raft_consensus.cc:504] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4ebe2e078d74794b44d139eda89a14f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46171 } } peers { permanent_uuid: "49078bc534c64097b2d54d69b011683f" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43225 } } peers { permanent_uuid: "230837dac3234917964f233c0db9da2a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39477 } }
I20260810 03:44:10.595683 9475 leader_election.cc:288] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49078bc534c64097b2d54d69b011683f (127.21.175.65:43225), 230837dac3234917964f233c0db9da2a (127.21.175.66:39477)
I20260810 03:44:10.599768 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.596326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:40769 (local address 127.21.175.66:39477)
0810 03:44:10.596484 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:10.596490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:10.596518 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:44:10.596576 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:10.596579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:10.596638 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:10.596713 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:10.597059 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.597668 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.598759 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.599188 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.599284 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.599322 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:10.599556 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 03:44:10.599634 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 03:44:10.599767 9451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.595935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:39477 (local address 127.21.175.67:40769)
0810 03:44:10.596060 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:10.596077 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:10.596175 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:10.596755 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:10.596758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:10.596778 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:10.596923 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:10.596931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.597786 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.597793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:10.598626 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:10.598636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.599317 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.599324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:10.599428 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.599452 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:44:10.599585 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 03:44:10.599654 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":4}
I20260810 03:44:10.599911 9465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.596071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:48791 (local address 127.21.175.65:43225)
0810 03:44:10.596213 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:10.596217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:10.596544 (+ 327us) server_negotiation.cc:407] Connection header received
0810 03:44:10.596668 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:10.596670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:10.596748 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:10.596831 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:10.597138 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.597748 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.598866 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.599293 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.599415 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.599460 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:10.599687 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 03:44:10.599774 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:44:10.599963 9476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.595935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:43225 (local address 127.21.175.67:48791)
0810 03:44:10.596430 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:10.596445 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:10.596567 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:10.596861 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:10.596863 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:10.596878 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:10.597019 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:10.597027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.597865 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.597868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:10.598743 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:10.598755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.599430 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.599433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:10.599545 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.599570 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:44:10.599667 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 03:44:10.599737 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":97,"thread_start_us":128,"threads_started":1}
I20260810 03:44:10.600523 9247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a78039367a475ebb682888672b982f" candidate_uuid: "f4ebe2e078d74794b44d139eda89a14f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "230837dac3234917964f233c0db9da2a" is_pre_election: true
I20260810 03:44:10.600541 9101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a78039367a475ebb682888672b982f" candidate_uuid: "f4ebe2e078d74794b44d139eda89a14f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49078bc534c64097b2d54d69b011683f" is_pre_election: true
I20260810 03:44:10.600863 9101 raft_consensus.cc:2454] T e5a78039367a475ebb682888672b982f P 49078bc534c64097b2d54d69b011683f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4ebe2e078d74794b44d139eda89a14f in term 0.
I20260810 03:44:10.600862 9247 raft_consensus.cc:2454] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4ebe2e078d74794b44d139eda89a14f in term 0.
I20260810 03:44:10.601387 9322 leader_election.cc:302] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49078bc534c64097b2d54d69b011683f, f4ebe2e078d74794b44d139eda89a14f; no voters:
I20260810 03:44:10.601717 9475 raft_consensus.cc:2781] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:44:10.601807 9475 raft_consensus.cc:482] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:44:10.601856 9475 raft_consensus.cc:3037] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:10.602918 9475 raft_consensus.cc:504] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4ebe2e078d74794b44d139eda89a14f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46171 } } peers { permanent_uuid: "49078bc534c64097b2d54d69b011683f" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43225 } } peers { permanent_uuid: "230837dac3234917964f233c0db9da2a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39477 } }
I20260810 03:44:10.603278 9475 leader_election.cc:288] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [CANDIDATE]: Term 1 election: Requested vote from peers 49078bc534c64097b2d54d69b011683f (127.21.175.65:43225), 230837dac3234917964f233c0db9da2a (127.21.175.66:39477)
I20260810 03:44:10.603730 9101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a78039367a475ebb682888672b982f" candidate_uuid: "f4ebe2e078d74794b44d139eda89a14f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49078bc534c64097b2d54d69b011683f"
I20260810 03:44:10.603770 9247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a78039367a475ebb682888672b982f" candidate_uuid: "f4ebe2e078d74794b44d139eda89a14f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "230837dac3234917964f233c0db9da2a"
I20260810 03:44:10.603875 9101 raft_consensus.cc:3037] T e5a78039367a475ebb682888672b982f P 49078bc534c64097b2d54d69b011683f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:10.603946 9247 raft_consensus.cc:3037] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:10.604956 9101 raft_consensus.cc:2454] T e5a78039367a475ebb682888672b982f P 49078bc534c64097b2d54d69b011683f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4ebe2e078d74794b44d139eda89a14f in term 1.
I20260810 03:44:10.605099 9247 raft_consensus.cc:2454] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4ebe2e078d74794b44d139eda89a14f in term 1.
I20260810 03:44:10.605301 9322 leader_election.cc:302] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49078bc534c64097b2d54d69b011683f, f4ebe2e078d74794b44d139eda89a14f; no voters:
I20260810 03:44:10.605509 9475 raft_consensus.cc:2781] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:10.605758 9475 raft_consensus.cc:686] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [term 1 LEADER]: Becoming Leader. State: Replica: f4ebe2e078d74794b44d139eda89a14f, State: Running, Role: LEADER
I20260810 03:44:10.606104 9475 consensus_queue.cc:237] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4ebe2e078d74794b44d139eda89a14f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46171 } } peers { permanent_uuid: "49078bc534c64097b2d54d69b011683f" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43225 } } peers { permanent_uuid: "230837dac3234917964f233c0db9da2a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39477 } }
I20260810 03:44:10.609262 8940 catalog_manager.cc:5495] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f reported cstate change: term changed from 0 to 1, leader changed from <none> to f4ebe2e078d74794b44d139eda89a14f (127.21.175.67). New cstate: current_term: 1 leader_uuid: "f4ebe2e078d74794b44d139eda89a14f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4ebe2e078d74794b44d139eda89a14f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 46171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49078bc534c64097b2d54d69b011683f" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "230837dac3234917964f233c0db9da2a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39477 } health_report { overall_health: UNKNOWN } } }
I20260810 03:44:10.649876 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.646399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.21.175.67:46171)
0810 03:44:10.646588 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:10.646591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:10.646604 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:44:10.646681 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:10.646683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:10.646739 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:10.646832 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:44:10.647055 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.647598 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.648422 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:10.648681 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:10.648766 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:10.649032 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:44:10.649454 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:44:10.649629 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:10.649694 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 03:44:10.649742 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:44:10.655891 9101 raft_consensus.cc:1261] T e5a78039367a475ebb682888672b982f P 49078bc534c64097b2d54d69b011683f [term 1 FOLLOWER]: Refusing update from remote peer f4ebe2e078d74794b44d139eda89a14f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:44:10.655900 9247 raft_consensus.cc:1261] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a [term 1 FOLLOWER]: Refusing update from remote peer f4ebe2e078d74794b44d139eda89a14f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:44:10.656674 9475 consensus_queue.cc:1035] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [LEADER]: Connected to new peer: Peer: permanent_uuid: "230837dac3234917964f233c0db9da2a" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 39477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:44:10.656916 9478 consensus_queue.cc:1035] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [LEADER]: Connected to new peer: Peer: permanent_uuid: "49078bc534c64097b2d54d69b011683f" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:44:10.664048 9482 mvcc.cc:205] Tried to move back new op lower bound from 7316821813879042048 to 7316821813685239808. Current Snapshot: MvccSnapshot[applied={T|T < 7316821813879042048}]
I20260810 03:44:10.673100 8937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:44:10.673302 8937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:44:10.676137 8937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a0ed26705d548f4906df42f2d9aa6c2: Sending DeleteTablet for 3 replicas of tablet e5a78039367a475ebb682888672b982f
I20260810 03:44:10.677146 9373 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a78039367a475ebb682888672b982f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:10 UTC) from {username='slave'} at 127.0.0.1:50600
I20260810 03:44:10.677167 9078 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a78039367a475ebb682888672b982f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:10 UTC) from {username='slave'} at 127.0.0.1:34044
I20260810 03:44:10.677266 9225 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a78039367a475ebb682888672b982f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:10 UTC) from {username='slave'} at 127.0.0.1:35990
I20260810 03:44:10.677651 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 9009
I20260810 03:44:10.677738 9494 tablet_replica.cc:330] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f: stopping tablet replica
I20260810 03:44:10.678082 9494 raft_consensus.cc:2229] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:44:10.678490 9494 raft_consensus.cc:2258] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:44:10.678861 9495 tablet_replica.cc:330] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a: stopping tablet replica
I20260810 03:44:10.679153 9495 raft_consensus.cc:2229] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:44:10.679451 9495 raft_consensus.cc:2258] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:44:10.681247 9495 ts_tablet_manager.cc:1905] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:44:10.684346 9495 ts_tablet_manager.cc:1918] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:44:10.684464 9495 log.cc:1196] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/e5a78039367a475ebb682888672b982f
I20260810 03:44:10.684876 9495 ts_tablet_manager.cc:1939] T e5a78039367a475ebb682888672b982f P 230837dac3234917964f233c0db9da2a: Deleting consensus metadata
I20260810 03:44:10.686266 8920 catalog_manager.cc:4841] TS 230837dac3234917964f233c0db9da2a (127.21.175.66:39477): tablet e5a78039367a475ebb682888672b982f (table test-table [id=6eba6aad2a9a41fd8ae977e4291d2d81]) successfully deleted
I20260810 03:44:10.687572 9494 ts_tablet_manager.cc:1905] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:44:10.689810 9494 ts_tablet_manager.cc:1918] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:44:10.689929 9494 log.cc:1196] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/e5a78039367a475ebb682888672b982f
I20260810 03:44:10.690227 9494 ts_tablet_manager.cc:1939] T e5a78039367a475ebb682888672b982f P f4ebe2e078d74794b44d139eda89a14f: Deleting consensus metadata
I20260810 03:44:10.691918 8920 catalog_manager.cc:4841] TS f4ebe2e078d74794b44d139eda89a14f (127.21.175.67:46171): tablet e5a78039367a475ebb682888672b982f (table test-table [id=6eba6aad2a9a41fd8ae977e4291d2d81]) successfully deleted
W20260810 03:44:10.693459 8919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.175.65:43225 (error 108)
I20260810 03:44:10.694437 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 9158
I20260810 03:44:10.694643 9464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:10.694312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:43225 (local address 127.0.0.1:34056)
0810 03:44:10.694474 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:10.694554 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:43225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260810 03:44:10.701747 8919 catalog_manager.cc:4584] TS 49078bc534c64097b2d54d69b011683f (127.21.175.65:43225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e5a78039367a475ebb682888672b982f: Network error: Client connection negotiation failed: client connection to 127.21.175.65:43225: connect: Connection refused (error 111)
I20260810 03:44:10.706558 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 9306
I20260810 03:44:10.714023 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 8901
2026-08-10T03:44:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1181 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:10 dist-test-slave-ss4n krb5kdc[9501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:10 dist-test-slave-ss4n krb5kdc[9501](info): set up 2 sockets
Aug 10 03:44:10 dist-test-slave-ss4n krb5kdc[9501](info): commencing operation
krb5kdc: starting...
W20260810 03:44:11.887260 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:11 dist-test-slave-ss4n krb5kdc[9501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:44:12.033988 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:46709
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:46709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:12.150269 9516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:12.150556 9516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:12.150599 9516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:12.154758 9516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:12.154827 9516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:12.154846 9516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:12.154863 9516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:12.154878 9516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:12.160282 9516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:46709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:46709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:12.161589 9516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:12.162678 9516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:12.171670 9523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:12.171876 9516 server_base.cc:1034] running on GCE node
W20260810 03:44:12.171744 9522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:12.171670 9526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:12.172670 9516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:12.173882 9516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:12.175105 9516 hybrid_clock.cc:648] HybridClock initialized: now 1786333452175056 us; error 47 us; skew 500 ppm
Aug 10 03:44:12 dist-test-slave-ss4n krb5kdc[9501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333452, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:12.178097 9516 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:44:12.178959 9516 webserver.cc:460] Webserver started at http://127.21.175.126:38217/ using document root <none> and password file <none>
I20260810 03:44:12.179490 9516 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:12.179550 9516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:12.179718 9516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:12.181869 9516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "0e621d67f0f9432b8d8ef738513a949f"
format_stamp: "Formatted at 2026-08-10 03:44:12 on dist-test-slave-ss4n"
server_key: "6702c97d8a9c19bda27ef24a92e0aed6"
server_key_iv: "2d327a0dbeb3d9fbf277fb39ffccd2dc"
server_key_version: "encryptionkey@0"
I20260810 03:44:12.182417 9516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "0e621d67f0f9432b8d8ef738513a949f"
format_stamp: "Formatted at 2026-08-10 03:44:12 on dist-test-slave-ss4n"
server_key: "6702c97d8a9c19bda27ef24a92e0aed6"
server_key_iv: "2d327a0dbeb3d9fbf277fb39ffccd2dc"
server_key_version: "encryptionkey@0"
I20260810 03:44:12.186115 9516 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:44:12.188486 9533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:12.189836 9516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:44:12.189946 9516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "0e621d67f0f9432b8d8ef738513a949f"
format_stamp: "Formatted at 2026-08-10 03:44:12 on dist-test-slave-ss4n"
server_key: "6702c97d8a9c19bda27ef24a92e0aed6"
server_key_iv: "2d327a0dbeb3d9fbf277fb39ffccd2dc"
server_key_version: "encryptionkey@0"
I20260810 03:44:12.190038 9516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:12.203125 9516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:12.206564 9516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:12.206727 9516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:12.213990 9516 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:46709
I20260810 03:44:12.214021 9594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:46709 every 8 connection(s)
I20260810 03:44:12.214998 9516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:12.217957 9596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:12.221974 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 9516
I20260810 03:44:12.222122 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:12.222476 22205 external_mini_cluster.cc:1428] Setting key 4d28e357a0b633978854d860b8ca84fc
I20260810 03:44:12.224498 9596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f: Bootstrap starting.
I20260810 03:44:12.226871 9596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:12.227646 9596 log.cc:824] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:12.229671 9596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f: No bootstrap required, opened a new log
Aug 10 03:44:12 dist-test-slave-ss4n krb5kdc[9501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:12.233595 9596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e621d67f0f9432b8d8ef738513a949f" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46709 } }
I20260810 03:44:12.233866 9596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:12.233951 9596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e621d67f0f9432b8d8ef738513a949f, State: Initialized, Role: FOLLOWER
I20260810 03:44:12.234463 9596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e621d67f0f9432b8d8ef738513a949f" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46709 } }
I20260810 03:44:12.234637 9596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:12.234730 9596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:12.234835 9596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:12.235765 9596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e621d67f0f9432b8d8ef738513a949f" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46709 } }
I20260810 03:44:12.236125 9596 leader_election.cc:302] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e621d67f0f9432b8d8ef738513a949f; no voters:
I20260810 03:44:12.236483 9596 leader_election.cc:288] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:12.236598 9602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:12.236915 9602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [term 1 LEADER]: Becoming Leader. State: Replica: 0e621d67f0f9432b8d8ef738513a949f, State: Running, Role: LEADER
I20260810 03:44:12.237329 9602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e621d67f0f9432b8d8ef738513a949f" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46709 } }
I20260810 03:44:12.237864 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:12.224125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.21.175.126:46709)
0810 03:44:12.224652 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:12.224675 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:12.224722 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:44:12.225384 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:12.225401 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:12.225784 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:12.226169 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:12.227007 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:12.228091 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:12.228996 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:12.229542 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:12.229719 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:12.232270 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:12.232275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:12.232290 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:12.232319 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:12.234426 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:12.235028 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:12.235033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:12.235037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:12.235154 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:12.235480 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:12.235485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:12.235491 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:12.235917 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:12.236144 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:12.236486 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 03:44:12.236835 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":143,"threads_started":1}
I20260810 03:44:12.237922 9596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:12.238591 9602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e621d67f0f9432b8d8ef738513a949f. Latest consensus state: current_term: 1 leader_uuid: "0e621d67f0f9432b8d8ef738513a949f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e621d67f0f9432b8d8ef738513a949f" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46709 } } }
I20260810 03:44:12.238713 9602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:12.239143 9609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:12.239300 9603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e621d67f0f9432b8d8ef738513a949f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e621d67f0f9432b8d8ef738513a949f" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46709 } } }
I20260810 03:44:12.239459 9603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:12.242172 9609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:12.247541 9609 catalog_manager.cc:1347] Generated new cluster ID: 2805882ee2fa4691b487d951fe4e01b4
I20260810 03:44:12.247612 9609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:12.267263 9609 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:12.268648 9609 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:12.280122 9609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f: Generated new TSK 0
I20260810 03:44:12.280997 9609 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:44:12.321235 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46709
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:12.451081 9623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:12.451436 9623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:12.451527 9623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:12.455803 9623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:12.455909 9623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:12.456007 9623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:44:12.461400 9623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:12.462579 9623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:12.463629 9623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:12.474059 9632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:12.474059 9635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:12.474066 9633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:12.474390 9623 server_base.cc:1034] running on GCE node
I20260810 03:44:12.474790 9623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:12.475411 9623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:12.476604 9623 hybrid_clock.cc:648] HybridClock initialized: now 1786333452476573 us; error 63 us; skew 500 ppm
Aug 10 03:44:12 dist-test-slave-ss4n krb5kdc[9501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333452, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:12.479697 9623 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:44:12.480679 9623 webserver.cc:460] Webserver started at http://127.21.175.65:35589/ using document root <none> and password file <none>
I20260810 03:44:12.481263 9623 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:12.481326 9623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:12.481647 9623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:12.483801 9623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "e532f564571d489d9e65b5943acaf219"
format_stamp: "Formatted at 2026-08-10 03:44:12 on dist-test-slave-ss4n"
server_key: "866ef6fb69ac289e167c07924e1ff92f"
server_key_iv: "2fc2a986db0c752b20582e8240f752f1"
server_key_version: "encryptionkey@0"
I20260810 03:44:12.484392 9623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "e532f564571d489d9e65b5943acaf219"
format_stamp: "Formatted at 2026-08-10 03:44:12 on dist-test-slave-ss4n"
server_key: "866ef6fb69ac289e167c07924e1ff92f"
server_key_iv: "2fc2a986db0c752b20582e8240f752f1"
server_key_version: "encryptionkey@0"
I20260810 03:44:12.488317 9623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 03:44:12.490880 9642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:12.492244 9623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 03:44:12.492385 9623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "e532f564571d489d9e65b5943acaf219"
format_stamp: "Formatted at 2026-08-10 03:44:12 on dist-test-slave-ss4n"
server_key: "866ef6fb69ac289e167c07924e1ff92f"
server_key_iv: "2fc2a986db0c752b20582e8240f752f1"
server_key_version: "encryptionkey@0"
I20260810 03:44:12.492501 9623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:12.507911 9623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:12.511498 9623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:12.511718 9623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:12.512351 9623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:12.513254 9623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:12.513329 9623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:12.513406 9623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:12.513449 9623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:12.523684 9623 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:44601
I20260810 03:44:12.523764 9765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:44601 every 8 connection(s)
I20260810 03:44:12.524696 9623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:44:12.530642 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 9623
I20260810 03:44:12.530803 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:44:12.531163 22205 external_mini_cluster.cc:1428] Setting key ac44dcd1438602b43c562db86435d305
Aug 10 03:44:12 dist-test-slave-ss4n krb5kdc[9501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333452, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:12.540195 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:12.526700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:33715 (local address 127.21.175.126:46709)
0810 03:44:12.526851 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:12.526855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:12.527773 (+ 918us) server_negotiation.cc:407] Connection header received
0810 03:44:12.529154 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:12.529158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:12.529206 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:12.529284 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:12.530617 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:12.531451 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:12.532892 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:12.533264 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:12.533397 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:12.536220 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:12.536223 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:12.536225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:12.536250 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:12.537817 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:12.538383 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:12.538387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:12.538389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:12.538476 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:12.538949 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:12.538955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:12.538960 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:12.539188 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:12.539299 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:12.539875 (+ 576us) server_negotiation.cc:299] Negotiation successful
0810 03:44:12.540023 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 03:44:12.541123 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:12.527092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46709 (local address 127.21.175.65:33715)
0810 03:44:12.527554 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:12.527651 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:12.528937 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:12.529463 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:12.529470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:12.529941 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:12.530432 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:12.530445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:12.531653 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:12.531658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:12.532736 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:12.532749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:12.533435 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:12.533438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:12.533697 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:12.534369 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:12.534393 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:12.536020 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:12.537948 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:12.537955 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:12.537973 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:12.538269 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:12.538609 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:12.538612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:12.538614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:12.538828 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:12.539314 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:12.539322 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:12.539611 (+ 289us) client_negotiation.cc:769] Sending connection context
0810 03:44:12.539832 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 03:44:12.540210 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
I20260810 03:44:12.542695 9766 heartbeater.cc:347] Connected to a master server at 127.21.175.126:46709
I20260810 03:44:12.542981 9766 heartbeater.cc:464] Registering TS with master...
I20260810 03:44:12.543521 9766 heartbeater.cc:511] Master 127.21.175.126:46709 requested a full tablet report, sending...
I20260810 03:44:12.545202 9553 ts_manager.cc:194] Registered new tserver with Master: e532f564571d489d9e65b5943acaf219 (127.21.175.65:44601)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
I20260810 03:44:12.546936 9553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:33715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:44:12.565305 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46709
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:12.682605 9772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:12.682880 9772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:12.682925 9772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:12.686991 9772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:12.687088 9772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:12.687150 9772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:44:12.692898 9772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:12.694175 9772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:12.695218 9772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:12.706043 9778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:12.706043 9780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:12.706305 9772 server_base.cc:1034] running on GCE node
W20260810 03:44:12.706486 9777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:12.706969 9772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:12.707726 9772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:12.708945 9772 hybrid_clock.cc:648] HybridClock initialized: now 1786333452708912 us; error 47 us; skew 500 ppm
Aug 10 03:44:12 dist-test-slave-ss4n krb5kdc[9501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333452, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:12.712255 9772 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:44:12.713358 9772 webserver.cc:460] Webserver started at http://127.21.175.66:42511/ using document root <none> and password file <none>
I20260810 03:44:12.714102 9772 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:12.714207 9772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:12.714483 9772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:12.717180 9772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "330d42efa8104c8a9419b27ab3d65cf8"
format_stamp: "Formatted at 2026-08-10 03:44:12 on dist-test-slave-ss4n"
server_key: "eaeb2a96e6c845c73e1f315601c17d98"
server_key_iv: "efe5a2fd8dd5b5ca3a35045a78a55cb9"
server_key_version: "encryptionkey@0"
I20260810 03:44:12.717862 9772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "330d42efa8104c8a9419b27ab3d65cf8"
format_stamp: "Formatted at 2026-08-10 03:44:12 on dist-test-slave-ss4n"
server_key: "eaeb2a96e6c845c73e1f315601c17d98"
server_key_iv: "efe5a2fd8dd5b5ca3a35045a78a55cb9"
server_key_version: "encryptionkey@0"
I20260810 03:44:12.722241 9772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:44:12.725062 9788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:12.726645 9772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:44:12.726799 9772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "330d42efa8104c8a9419b27ab3d65cf8"
format_stamp: "Formatted at 2026-08-10 03:44:12 on dist-test-slave-ss4n"
server_key: "eaeb2a96e6c845c73e1f315601c17d98"
server_key_iv: "efe5a2fd8dd5b5ca3a35045a78a55cb9"
server_key_version: "encryptionkey@0"
I20260810 03:44:12.726933 9772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:12.743165 9772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:12.746881 9772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:12.747117 9772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:12.747768 9772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:12.748682 9772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:12.748757 9772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:12.748833 9772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:12.748888 9772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:12.758378 9772 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:41271
I20260810 03:44:12.758402 9923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:41271 every 8 connection(s)
I20260810 03:44:12.759444 9772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:44:12.763154 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 9772
I20260810 03:44:12.763257 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:44:12.763548 22205 external_mini_cluster.cc:1428] Setting key c0c100bccce26fed14351b7c2beb57b2
Aug 10 03:44:12 dist-test-slave-ss4n krb5kdc[9501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333452, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
I20260810 03:44:12.775784 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:12.761389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:34069 (local address 127.21.175.126:46709)
0810 03:44:12.761590 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:12.761594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:12.762491 (+ 897us) server_negotiation.cc:407] Connection header received
0810 03:44:12.764072 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:12.764075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:12.764154 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:12.764252 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:12.765673 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:12.766215 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:12.767673 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:12.767902 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:12.767987 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:12.771541 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:12.771548 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:12.771555 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:12.771593 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:12.773386 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:12.774024 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:12.774029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:12.774032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:12.774131 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:12.774496 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:12.774502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:12.774505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:12.774772 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:12.774890 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:12.775415 (+ 525us) server_negotiation.cc:299] Negotiation successful
0810 03:44:12.775566 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:44:12.776695 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:12.761842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46709 (local address 127.21.175.66:34069)
0810 03:44:12.762327 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:12.762373 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:12.763812 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:12.764440 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:12.764448 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:12.764878 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:12.765459 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:12.765473 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:12.766358 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:12.766361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:12.767485 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:12.767501 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:12.768078 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:12.768083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:12.768321 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:12.769018 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:12.769062 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:12.771317 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:12.773579 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:12.773610 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:12.773623 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:12.773893 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:12.774230 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:12.774232 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:12.774234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:12.774373 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:12.774888 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:12.774893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:12.775153 (+ 260us) client_negotiation.cc:769] Sending connection context
0810 03:44:12.775309 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 03:44:12.775682 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":119,"threads_started":1}
I20260810 03:44:12.778244 9924 heartbeater.cc:347] Connected to a master server at 127.21.175.126:46709
I20260810 03:44:12.778503 9924 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
I20260810 03:44:12.779179 9924 heartbeater.cc:511] Master 127.21.175.126:46709 requested a full tablet report, sending...
I20260810 03:44:12.780580 9553 ts_manager.cc:194] Registered new tserver with Master: 330d42efa8104c8a9419b27ab3d65cf8 (127.21.175.66:41271)
I20260810 03:44:12.781508 9553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:34069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:44:12.800782 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46709
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:12.918223 9929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:12.918510 9929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:12.918556 9929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:12.922744 9929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:12.922816 9929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:12.922876 9929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:44:12.928965 9929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:12.930171 9929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:12.931303 9929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:12.941988 9935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:12.941951 9934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:12.941951 9938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:12.942332 9929 server_base.cc:1034] running on GCE node
I20260810 03:44:12.942718 9929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:12.943425 9929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:12.944612 9929 hybrid_clock.cc:648] HybridClock initialized: now 1786333452944596 us; error 28 us; skew 500 ppm
Aug 10 03:44:12 dist-test-slave-ss4n krb5kdc[9501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333452, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:12.947759 9929 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:44:12.948709 9929 webserver.cc:460] Webserver started at http://127.21.175.67:36123/ using document root <none> and password file <none>
I20260810 03:44:12.949298 9929 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:12.949404 9929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:12.949671 9929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:12.951815 9929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "2a7f21a562494ba89250f96f1f8fc339"
format_stamp: "Formatted at 2026-08-10 03:44:12 on dist-test-slave-ss4n"
server_key: "9e4a18345a250eb416c3cebf59f26070"
server_key_iv: "9e401dc175884970cb4b205ca9a27643"
server_key_version: "encryptionkey@0"
I20260810 03:44:12.952405 9929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "2a7f21a562494ba89250f96f1f8fc339"
format_stamp: "Formatted at 2026-08-10 03:44:12 on dist-test-slave-ss4n"
server_key: "9e4a18345a250eb416c3cebf59f26070"
server_key_iv: "9e401dc175884970cb4b205ca9a27643"
server_key_version: "encryptionkey@0"
I20260810 03:44:12.956315 9929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:44:12.958871 9947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:12.960320 9929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:44:12.960418 9929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "2a7f21a562494ba89250f96f1f8fc339"
format_stamp: "Formatted at 2026-08-10 03:44:12 on dist-test-slave-ss4n"
server_key: "9e4a18345a250eb416c3cebf59f26070"
server_key_iv: "9e401dc175884970cb4b205ca9a27643"
server_key_version: "encryptionkey@0"
I20260810 03:44:12.960564 9929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:12.995911 9929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:13.000294 9929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:13.000510 9929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:13.001309 9929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:13.002275 9929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:13.002326 9929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:13.002369 9929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:13.002384 9929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:13.011821 9929 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:34023
I20260810 03:44:13.011843 10074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:34023 every 8 connection(s)
I20260810 03:44:13.012853 9929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:44:13.019258 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 9929
I20260810 03:44:13.019450 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:44:13.019788 22205 external_mini_cluster.cc:1428] Setting key b460321e700f249e3ce9e49573d84a5a
Aug 10 03:44:13 dist-test-slave-ss4n krb5kdc[9501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333452, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:13.028903 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:13.014970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:33727 (local address 127.21.175.126:46709)
0810 03:44:13.015136 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:13.015140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:13.016102 (+ 962us) server_negotiation.cc:407] Connection header received
0810 03:44:13.017709 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:13.017713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:13.017781 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:13.017905 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:13.019444 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.020104 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.021840 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.022058 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.022200 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:13.025170 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:13.025173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:13.025175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:13.025200 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:13.026767 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:13.027317 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:13.027325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:13.027326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:13.027403 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:13.027771 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:13.027775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:13.027777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:13.027972 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:13.028092 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:13.028607 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 03:44:13.028741 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 03:44:13.029909 10077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:13.015347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46709 (local address 127.21.175.67:33727)
0810 03:44:13.015899 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:13.015949 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:13.017401 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:13.018115 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:13.018123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:13.018524 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:13.019207 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:13.019232 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.020284 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.020287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:13.021696 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:13.021708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.022274 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.022279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:13.022549 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:13.023173 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:13.023192 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:13.024973 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:13.026899 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:13.026905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:13.026922 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:13.027200 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:13.027498 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:13.027502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:13.027504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:13.027667 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:13.028083 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:13.028091 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:13.028359 (+ 268us) client_negotiation.cc:769] Sending connection context
0810 03:44:13.028561 (+ 202us) client_negotiation.cc:240] Negotiation successful
0810 03:44:13.028921 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":123,"threads_started":1}
I20260810 03:44:13.031203 10075 heartbeater.cc:347] Connected to a master server at 127.21.175.126:46709
I20260810 03:44:13.031438 10075 heartbeater.cc:464] Registering TS with master...
I20260810 03:44:13.032012 10075 heartbeater.cc:511] Master 127.21.175.126:46709 requested a full tablet report, sending...
I20260810 03:44:13.033123 9553 ts_manager.cc:194] Registered new tserver with Master: 2a7f21a562494ba89250f96f1f8fc339 (127.21.175.67:34023)
I20260810 03:44:13.033918 9553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:33727
I20260810 03:44:13.034377 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:44:13.046211 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:13.036628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44916 (local address 127.21.175.126:46709)
0810 03:44:13.036793 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:13.036797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:13.036974 (+ 177us) server_negotiation.cc:407] Connection header received
0810 03:44:13.037130 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:13.037133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:13.037179 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:13.037261 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:13.038091 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.038753 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.039901 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.040248 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.040349 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:13.042265 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:13.042268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:13.042270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:13.042295 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:13.043931 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:13.044652 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:13.044655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:13.044656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:13.044746 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:13.045183 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:13.045186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:13.045187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:13.045416 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:13.045642 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:13.045992 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 03:44:13.046085 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:44:13.049450 9553 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:44:13.052497 9553 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:44:13.068239 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:13.061841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.21.175.66:41271)
0810 03:44:13.062244 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:13.062256 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:13.062746 (+ 490us) server_negotiation.cc:407] Connection header received
0810 03:44:13.062826 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:13.062833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:13.063006 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:13.063159 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:13.063542 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.064553 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.066581 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.067465 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.067579 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:13.067732 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:13.067963 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 03:44:13.068081 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":103,"threads_started":1}
I20260810 03:44:13.068233 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:13.061392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:34023 (local address 127.0.0.1:46674)
0810 03:44:13.062091 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:13.062161 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:13.062288 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:13.063154 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:13.063159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:13.063181 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:13.063336 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:13.063344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.064820 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.064824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:13.066429 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:13.066441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.067801 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.067804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:13.067913 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:13.067941 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 03:44:13.068014 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 03:44:13.068080 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":140,"thread_start_us":144,"threads_started":1}
I20260810 03:44:13.068746 10093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:13.062520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.21.175.65:44601)
0810 03:44:13.062911 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:13.062918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:13.063170 (+ 252us) server_negotiation.cc:407] Connection header received
0810 03:44:13.063418 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:13.063425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:13.063626 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:13.063776 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:13.064036 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.065027 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.066779 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.067903 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.067977 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:13.068122 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:13.068478 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 03:44:13.068605 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":101,"threads_started":1}
I20260810 03:44:13.068950 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:13.061572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46674 (local address 127.21.175.67:34023)
0810 03:44:13.062233 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:13.062248 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:13.062277 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:44:13.062507 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:13.062513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:13.062820 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:13.062966 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:13.063464 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.064688 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.066584 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.067465 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.067585 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:13.067732 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:13.068702 (+ 970us) server_negotiation.cc:299] Negotiation successful
0810 03:44:13.068805 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":144,"threads_started":1}
I20260810 03:44:13.068235 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:13.061426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:41271 (local address 127.0.0.1:59712)
0810 03:44:13.062547 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:13.062562 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:13.062652 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:13.063312 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:13.063315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:13.063329 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:13.063406 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:13.063414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.064708 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.064714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:13.066429 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:13.066441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.067623 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.067627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:13.067734 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:13.067780 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 03:44:13.067910 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 03:44:13.068047 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1062,"mutex_wait_us":192,"thread_start_us":106,"threads_started":1}
I20260810 03:44:13.068615 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:13.062250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:44601 (local address 127.0.0.1:45692)
0810 03:44:13.063059 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:13.063072 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:13.063253 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:13.063771 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:13.063774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:13.063794 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:13.063887 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:13.063893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.065203 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.065207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:13.066542 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:13.066555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.068225 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.068228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:13.068325 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:13.068349 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:44:13.068426 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 03:44:13.068496 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":95,"thread_start_us":91,"threads_started":1}
I20260810 03:44:13.072086 9693 tablet_service.cc:1467] Processing CreateTablet for tablet 3ece7548fd674b60bc44f766a2060256 (DEFAULT_TABLE table=test-table [id=3cbb0b5924ed448b87aa1f5fa0ad686e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:13.072232 10000 tablet_service.cc:1467] Processing CreateTablet for tablet 3ece7548fd674b60bc44f766a2060256 (DEFAULT_TABLE table=test-table [id=3cbb0b5924ed448b87aa1f5fa0ad686e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:13.072229 9842 tablet_service.cc:1467] Processing CreateTablet for tablet 3ece7548fd674b60bc44f766a2060256 (DEFAULT_TABLE table=test-table [id=3cbb0b5924ed448b87aa1f5fa0ad686e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:13.073191 9693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ece7548fd674b60bc44f766a2060256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:13.073248 10000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ece7548fd674b60bc44f766a2060256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:13.073248 9842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ece7548fd674b60bc44f766a2060256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:13.079627 10094 tablet_bootstrap.cc:492] T 3ece7548fd674b60bc44f766a2060256 P e532f564571d489d9e65b5943acaf219: Bootstrap starting.
I20260810 03:44:13.081522 10095 tablet_bootstrap.cc:492] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339: Bootstrap starting.
I20260810 03:44:13.082535 10094 tablet_bootstrap.cc:654] T 3ece7548fd674b60bc44f766a2060256 P e532f564571d489d9e65b5943acaf219: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:13.083468 10094 log.cc:824] T 3ece7548fd674b60bc44f766a2060256 P e532f564571d489d9e65b5943acaf219: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:13.083847 10095 tablet_bootstrap.cc:654] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:13.084352 10097 tablet_bootstrap.cc:492] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8: Bootstrap starting.
I20260810 03:44:13.085124 10095 log.cc:824] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:13.086537 10097 tablet_bootstrap.cc:654] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:13.087436 10095 tablet_bootstrap.cc:492] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339: No bootstrap required, opened a new log
I20260810 03:44:13.087663 10095 ts_tablet_manager.cc:1397] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20260810 03:44:13.087814 10097 log.cc:824] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:13.088896 10094 tablet_bootstrap.cc:492] T 3ece7548fd674b60bc44f766a2060256 P e532f564571d489d9e65b5943acaf219: No bootstrap required, opened a new log
I20260810 03:44:13.089112 10094 ts_tablet_manager.cc:1397] T 3ece7548fd674b60bc44f766a2060256 P e532f564571d489d9e65b5943acaf219: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260810 03:44:13.089951 10097 tablet_bootstrap.cc:492] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8: No bootstrap required, opened a new log
I20260810 03:44:13.090145 10097 ts_tablet_manager.cc:1397] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:44:13.091735 10095 raft_consensus.cc:348] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7f21a562494ba89250f96f1f8fc339" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 34023 } } peers { permanent_uuid: "330d42efa8104c8a9419b27ab3d65cf8" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 41271 } } peers { permanent_uuid: "e532f564571d489d9e65b5943acaf219" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44601 } }
I20260810 03:44:13.092048 10095 raft_consensus.cc:374] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:13.092111 10095 raft_consensus.cc:729] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7f21a562494ba89250f96f1f8fc339, State: Initialized, Role: FOLLOWER
I20260810 03:44:13.092604 10095 consensus_queue.cc:260] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7f21a562494ba89250f96f1f8fc339" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 34023 } } peers { permanent_uuid: "330d42efa8104c8a9419b27ab3d65cf8" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 41271 } } peers { permanent_uuid: "e532f564571d489d9e65b5943acaf219" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44601 } }
I20260810 03:44:13.093322 10075 heartbeater.cc:503] Master 127.21.175.126:46709 was elected leader, sending a full tablet report...
I20260810 03:44:13.093434 10094 raft_consensus.cc:348] T 3ece7548fd674b60bc44f766a2060256 P e532f564571d489d9e65b5943acaf219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7f21a562494ba89250f96f1f8fc339" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 34023 } } peers { permanent_uuid: "330d42efa8104c8a9419b27ab3d65cf8" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 41271 } } peers { permanent_uuid: "e532f564571d489d9e65b5943acaf219" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44601 } }
I20260810 03:44:13.093722 10094 raft_consensus.cc:374] T 3ece7548fd674b60bc44f766a2060256 P e532f564571d489d9e65b5943acaf219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:13.093748 10095 ts_tablet_manager.cc:1428] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260810 03:44:13.093786 10094 raft_consensus.cc:729] T 3ece7548fd674b60bc44f766a2060256 P e532f564571d489d9e65b5943acaf219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e532f564571d489d9e65b5943acaf219, State: Initialized, Role: FOLLOWER
I20260810 03:44:13.094209 10094 consensus_queue.cc:260] T 3ece7548fd674b60bc44f766a2060256 P e532f564571d489d9e65b5943acaf219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7f21a562494ba89250f96f1f8fc339" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 34023 } } peers { permanent_uuid: "330d42efa8104c8a9419b27ab3d65cf8" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 41271 } } peers { permanent_uuid: "e532f564571d489d9e65b5943acaf219" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44601 } }
I20260810 03:44:13.094872 9766 heartbeater.cc:503] Master 127.21.175.126:46709 was elected leader, sending a full tablet report...
I20260810 03:44:13.095211 10094 ts_tablet_manager.cc:1428] T 3ece7548fd674b60bc44f766a2060256 P e532f564571d489d9e65b5943acaf219: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 03:44:13.095141 10097 raft_consensus.cc:348] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7f21a562494ba89250f96f1f8fc339" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 34023 } } peers { permanent_uuid: "330d42efa8104c8a9419b27ab3d65cf8" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 41271 } } peers { permanent_uuid: "e532f564571d489d9e65b5943acaf219" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44601 } }
I20260810 03:44:13.095381 10097 raft_consensus.cc:374] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:13.095438 10097 raft_consensus.cc:729] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 330d42efa8104c8a9419b27ab3d65cf8, State: Initialized, Role: FOLLOWER
I20260810 03:44:13.095862 10097 consensus_queue.cc:260] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7f21a562494ba89250f96f1f8fc339" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 34023 } } peers { permanent_uuid: "330d42efa8104c8a9419b27ab3d65cf8" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 41271 } } peers { permanent_uuid: "e532f564571d489d9e65b5943acaf219" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44601 } }
I20260810 03:44:13.097700 9924 heartbeater.cc:503] Master 127.21.175.126:46709 was elected leader, sending a full tablet report...
I20260810 03:44:13.098628 10097 ts_tablet_manager.cc:1428] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260810 03:44:13.260703 9925 tablet.cc:2369] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:44:13.263958 10076 tablet.cc:2369] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:44:13.276172 9767 tablet.cc:2369] T 3ece7548fd674b60bc44f766a2060256 P e532f564571d489d9e65b5943acaf219: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:44:13.388753 10104 raft_consensus.cc:482] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:44:13.388979 10104 raft_consensus.cc:504] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7f21a562494ba89250f96f1f8fc339" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 34023 } } peers { permanent_uuid: "330d42efa8104c8a9419b27ab3d65cf8" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 41271 } } peers { permanent_uuid: "e532f564571d489d9e65b5943acaf219" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44601 } }
I20260810 03:44:13.390233 10104 leader_election.cc:288] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a7f21a562494ba89250f96f1f8fc339 (127.21.175.67:34023), e532f564571d489d9e65b5943acaf219 (127.21.175.65:44601)
I20260810 03:44:13.395962 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:13.390720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:49765 (local address 127.21.175.67:34023)
0810 03:44:13.390888 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:13.390894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:13.391025 (+ 131us) server_negotiation.cc:407] Connection header received
0810 03:44:13.391411 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:13.391415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:13.391476 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:13.391655 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:13.391990 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.392633 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.394647 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.395250 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.395377 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:13.395431 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:13.395786 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 03:44:13.395856 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 03:44:13.396104 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:13.390616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:34023 (local address 127.21.175.66:49765)
0810 03:44:13.390914 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:13.390927 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:13.391070 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:13.391695 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:13.391701 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:13.391724 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:13.391866 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:13.391873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.392758 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.392765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:13.394461 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:13.394472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.395398 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.395402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:13.395597 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:13.395653 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 03:44:13.395751 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 03:44:13.395896 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":192,"thread_start_us":134,"threads_started":1}
I20260810 03:44:13.396108 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:13.391121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:44601 (local address 127.21.175.66:34227)
0810 03:44:13.391474 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:13.391489 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:13.391613 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:13.391895 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:13.391897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:13.391913 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:13.392082 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:13.392089 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.392862 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.392866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:13.394460 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:13.394472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.395398 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.395402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:13.395656 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:13.395677 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 03:44:13.395751 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 03:44:13.395894 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":190,"thread_start_us":81,"threads_started":1}
I20260810 03:44:13.396492 10093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:13.391276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:34227 (local address 127.21.175.65:44601)
0810 03:44:13.391408 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:13.391411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:13.391575 (+ 164us) server_negotiation.cc:407] Connection header received
0810 03:44:13.391728 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:13.391732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:13.391776 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:13.391862 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:13.392202 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.392778 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.394608 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.395250 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.395377 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:13.395431 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:13.396290 (+ 859us) server_negotiation.cc:299] Negotiation successful
0810 03:44:13.396382 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 03:44:13.397073 9716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ece7548fd674b60bc44f766a2060256" candidate_uuid: "330d42efa8104c8a9419b27ab3d65cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e532f564571d489d9e65b5943acaf219" is_pre_election: true
I20260810 03:44:13.397094 10024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ece7548fd674b60bc44f766a2060256" candidate_uuid: "330d42efa8104c8a9419b27ab3d65cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7f21a562494ba89250f96f1f8fc339" is_pre_election: true
I20260810 03:44:13.397351 9716 raft_consensus.cc:2454] T 3ece7548fd674b60bc44f766a2060256 P e532f564571d489d9e65b5943acaf219 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 330d42efa8104c8a9419b27ab3d65cf8 in term 0.
I20260810 03:44:13.397535 10024 raft_consensus.cc:2454] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 330d42efa8104c8a9419b27ab3d65cf8 in term 0.
I20260810 03:44:13.397979 9791 leader_election.cc:302] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 330d42efa8104c8a9419b27ab3d65cf8, e532f564571d489d9e65b5943acaf219; no voters:
I20260810 03:44:13.398299 10104 raft_consensus.cc:2781] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:44:13.398382 10104 raft_consensus.cc:482] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:44:13.398413 10104 raft_consensus.cc:3037] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:13.399582 10104 raft_consensus.cc:504] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7f21a562494ba89250f96f1f8fc339" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 34023 } } peers { permanent_uuid: "330d42efa8104c8a9419b27ab3d65cf8" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 41271 } } peers { permanent_uuid: "e532f564571d489d9e65b5943acaf219" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44601 } }
I20260810 03:44:13.399940 10104 leader_election.cc:288] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [CANDIDATE]: Term 1 election: Requested vote from peers 2a7f21a562494ba89250f96f1f8fc339 (127.21.175.67:34023), e532f564571d489d9e65b5943acaf219 (127.21.175.65:44601)
I20260810 03:44:13.400484 10024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ece7548fd674b60bc44f766a2060256" candidate_uuid: "330d42efa8104c8a9419b27ab3d65cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7f21a562494ba89250f96f1f8fc339"
I20260810 03:44:13.400537 9716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ece7548fd674b60bc44f766a2060256" candidate_uuid: "330d42efa8104c8a9419b27ab3d65cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e532f564571d489d9e65b5943acaf219"
I20260810 03:44:13.400681 9716 raft_consensus.cc:3037] T 3ece7548fd674b60bc44f766a2060256 P e532f564571d489d9e65b5943acaf219 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:13.400681 10024 raft_consensus.cc:3037] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:13.402329 10024 raft_consensus.cc:2454] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 330d42efa8104c8a9419b27ab3d65cf8 in term 1.
I20260810 03:44:13.402329 9716 raft_consensus.cc:2454] T 3ece7548fd674b60bc44f766a2060256 P e532f564571d489d9e65b5943acaf219 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 330d42efa8104c8a9419b27ab3d65cf8 in term 1.
I20260810 03:44:13.402683 9791 leader_election.cc:302] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 330d42efa8104c8a9419b27ab3d65cf8, e532f564571d489d9e65b5943acaf219; no voters:
I20260810 03:44:13.402877 10104 raft_consensus.cc:2781] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:13.403190 10104 raft_consensus.cc:686] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [term 1 LEADER]: Becoming Leader. State: Replica: 330d42efa8104c8a9419b27ab3d65cf8, State: Running, Role: LEADER
I20260810 03:44:13.403565 10104 consensus_queue.cc:237] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7f21a562494ba89250f96f1f8fc339" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 34023 } } peers { permanent_uuid: "330d42efa8104c8a9419b27ab3d65cf8" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 41271 } } peers { permanent_uuid: "e532f564571d489d9e65b5943acaf219" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44601 } }
I20260810 03:44:13.407135 9552 catalog_manager.cc:5495] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 330d42efa8104c8a9419b27ab3d65cf8 (127.21.175.66). New cstate: current_term: 1 leader_uuid: "330d42efa8104c8a9419b27ab3d65cf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7f21a562494ba89250f96f1f8fc339" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 34023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "330d42efa8104c8a9419b27ab3d65cf8" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 41271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e532f564571d489d9e65b5943acaf219" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44601 } health_report { overall_health: UNKNOWN } } }
I20260810 03:44:13.436600 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:13.432188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59714 (local address 127.21.175.66:41271)
0810 03:44:13.432318 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:13.432321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:13.432335 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:44:13.432534 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:13.432536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:13.432601 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:13.432697 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:44:13.433006 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.433636 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.434682 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.434958 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.435072 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:13.435478 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:44:13.436042 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:44:13.436190 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:13.436414 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 03:44:13.436478 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:44:13.443475 10024 raft_consensus.cc:1261] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339 [term 1 FOLLOWER]: Refusing update from remote peer 330d42efa8104c8a9419b27ab3d65cf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:44:13.443534 9716 raft_consensus.cc:1261] T 3ece7548fd674b60bc44f766a2060256 P e532f564571d489d9e65b5943acaf219 [term 1 FOLLOWER]: Refusing update from remote peer 330d42efa8104c8a9419b27ab3d65cf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:44:13.444343 10104 consensus_queue.cc:1035] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e532f564571d489d9e65b5943acaf219" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 44601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:44:13.444650 10107 consensus_queue.cc:1035] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a7f21a562494ba89250f96f1f8fc339" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 34023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:44:13.453902 10111 mvcc.cc:205] Tried to move back new op lower bound from 7316821825296044032 to 7316821825144102912. Current Snapshot: MvccSnapshot[applied={T|T < 7316821825296044032}]
I20260810 03:44:13.462392 9551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:44:13.462556 9551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:44:13.465052 9551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e621d67f0f9432b8d8ef738513a949f: Sending DeleteTablet for 3 replicas of tablet 3ece7548fd674b60bc44f766a2060256
I20260810 03:44:13.465963 9842 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ece7548fd674b60bc44f766a2060256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:13 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:59712
I20260810 03:44:13.466118 9693 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ece7548fd674b60bc44f766a2060256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:13 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:45692
I20260810 03:44:13.466572 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 9623
I20260810 03:44:13.466876 10000 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ece7548fd674b60bc44f766a2060256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:13 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:46674
I20260810 03:44:13.468411 10122 tablet_replica.cc:330] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339: stopping tablet replica
I20260810 03:44:13.468969 10122 raft_consensus.cc:2229] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:44:13.469401 10122 raft_consensus.cc:2258] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:44:13.471441 10122 ts_tablet_manager.cc:1905] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:44:13.474646 10122 ts_tablet_manager.cc:1918] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:44:13.474768 10122 log.cc:1196] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/3ece7548fd674b60bc44f766a2060256
I20260810 03:44:13.475128 10122 ts_tablet_manager.cc:1939] T 3ece7548fd674b60bc44f766a2060256 P 2a7f21a562494ba89250f96f1f8fc339: Deleting consensus metadata
I20260810 03:44:13.476667 10120 tablet_replica.cc:330] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8: stopping tablet replica
I20260810 03:44:13.476696 9536 catalog_manager.cc:4841] TS 2a7f21a562494ba89250f96f1f8fc339 (127.21.175.67:34023): tablet 3ece7548fd674b60bc44f766a2060256 (table test-table [id=3cbb0b5924ed448b87aa1f5fa0ad686e]) successfully deleted
I20260810 03:44:13.477073 10120 raft_consensus.cc:2229] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:44:13.477634 10120 raft_consensus.cc:2258] T 3ece7548fd674b60bc44f766a2060256 P 330d42efa8104c8a9419b27ab3d65cf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:44:13.479220 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 9772
W20260810 03:44:13.479241 9536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.175.65:44601 (error 108)
I20260810 03:44:13.486488 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:13.486191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:44601 (local address 127.0.0.1:45700)
0810 03:44:13.486335 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:13.486406 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 03:44:13.486788 9536 catalog_manager.cc:4584] TS e532f564571d489d9e65b5943acaf219 (127.21.175.65:44601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ece7548fd674b60bc44f766a2060256: Network error: Client connection negotiation failed: client connection to 127.21.175.65:44601: connect: Connection refused (error 111)
I20260810 03:44:13.488467 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 9929
I20260810 03:44:13.488741 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:13.488471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:41271 (local address 127.0.0.1:59722)
0810 03:44:13.488592 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:13.488655 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.66:41271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 03:44:13.498919 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 9516
2026-08-10T03:44:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2787 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T03:44:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:13Z Disabled control of system clock
I20260810 03:44:13.536322 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:38175
--webserver_interface=127.21.175.126
--webserver_port=0
--builtin_ntp_servers=127.21.175.84:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:38175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:44:13.655023 10128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:13.655300 10128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:13.655335 10128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:13.659405 10128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:13.659468 10128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:13.659484 10128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:13.659500 10128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:13.659515 10128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:13.664351 10128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:38175
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:38175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:13.665515 10128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:13.667009 10128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:13.676194 10133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:13.676250 10128 server_base.cc:1034] running on GCE node
W20260810 03:44:13.676192 10135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:13.676186 10137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:13.677109 10128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:13.678352 10128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:13.679546 10128 hybrid_clock.cc:648] HybridClock initialized: now 1786333453679530 us; error 27 us; skew 500 ppm
I20260810 03:44:13.681627 10128 webserver.cc:460] Webserver started at http://127.21.175.126:39547/ using document root <none> and password file <none>
I20260810 03:44:13.682243 10128 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:13.682312 10128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:13.682549 10128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:13.684672 10128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "4c23945d1cd8463a9c1562f7b9293cb7"
format_stamp: "Formatted at 2026-08-10 03:44:13 on dist-test-slave-ss4n"
server_key: "262c60fa05a091e6380479023257bc66"
server_key_iv: "a7cb8a5a3565130542618606ab653069"
server_key_version: "encryptionkey@0"
I20260810 03:44:13.685277 10128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "4c23945d1cd8463a9c1562f7b9293cb7"
format_stamp: "Formatted at 2026-08-10 03:44:13 on dist-test-slave-ss4n"
server_key: "262c60fa05a091e6380479023257bc66"
server_key_iv: "a7cb8a5a3565130542618606ab653069"
server_key_version: "encryptionkey@0"
I20260810 03:44:13.689158 10128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 03:44:13.692294 10145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:13.693919 10128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 03:44:13.694082 10128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "4c23945d1cd8463a9c1562f7b9293cb7"
format_stamp: "Formatted at 2026-08-10 03:44:13 on dist-test-slave-ss4n"
server_key: "262c60fa05a091e6380479023257bc66"
server_key_iv: "a7cb8a5a3565130542618606ab653069"
server_key_version: "encryptionkey@0"
I20260810 03:44:13.694214 10128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:13.710242 10128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:13.711072 10128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:13.711294 10128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:13.718834 10128 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:38175
I20260810 03:44:13.718853 10205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:38175 every 8 connection(s)
I20260810 03:44:13.719887 10128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:13.723434 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 10128
I20260810 03:44:13.723589 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:13.724049 22205 external_mini_cluster.cc:1428] Setting key 0c064ad02f8abbcc122e5328187d964c
I20260810 03:44:13.724467 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:13.732482 10206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7: Bootstrap starting.
I20260810 03:44:13.735291 10206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:13.735903 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:13.725782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.21.175.126:38175)
0810 03:44:13.726628 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:13.726653 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:13.726705 (+ 52us) server_negotiation.cc:407] Connection header received
0810 03:44:13.727796 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:13.727819 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:13.728275 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:13.728623 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:13.728999 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.730214 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.731510 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:13.732198 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:13.732363 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:13.733579 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:13.733587 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:13.733608 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:13.734156 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:13.734226 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:13.734252 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:13.734356 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:13.734536 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 03:44:13.734819 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":188,"threads_started":1}
I20260810 03:44:13.736239 10206 log.cc:824] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:13.738860 10206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7: No bootstrap required, opened a new log
I20260810 03:44:13.743125 10206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c23945d1cd8463a9c1562f7b9293cb7" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38175 } }
I20260810 03:44:13.743386 10206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:13.743443 10206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c23945d1cd8463a9c1562f7b9293cb7, State: Initialized, Role: FOLLOWER
I20260810 03:44:13.743861 10206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c23945d1cd8463a9c1562f7b9293cb7" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38175 } }
I20260810 03:44:13.744012 10206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:13.744061 10206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:13.744148 10206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:13.745261 10206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c23945d1cd8463a9c1562f7b9293cb7" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38175 } }
I20260810 03:44:13.745698 10206 leader_election.cc:302] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c23945d1cd8463a9c1562f7b9293cb7; no voters:
I20260810 03:44:13.746093 10206 leader_election.cc:288] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:13.746251 10212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:13.746615 10212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [term 1 LEADER]: Becoming Leader. State: Replica: 4c23945d1cd8463a9c1562f7b9293cb7, State: Running, Role: LEADER
I20260810 03:44:13.746932 10212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c23945d1cd8463a9c1562f7b9293cb7" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38175 } }
I20260810 03:44:13.747694 10206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:13.749003 10214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c23945d1cd8463a9c1562f7b9293cb7. Latest consensus state: current_term: 1 leader_uuid: "4c23945d1cd8463a9c1562f7b9293cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c23945d1cd8463a9c1562f7b9293cb7" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38175 } } }
I20260810 03:44:13.749001 10213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c23945d1cd8463a9c1562f7b9293cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c23945d1cd8463a9c1562f7b9293cb7" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 38175 } } }
I20260810 03:44:13.749212 10214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:13.749248 10213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:13.750627 10220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:13.754350 10220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:13.761186 10220 catalog_manager.cc:1347] Generated new cluster ID: f45b13b8b0cd45e3b93c8001f83a69c9
I20260810 03:44:13.761289 10220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:13.796703 10220 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:13.798666 10220 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:13.810395 10220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7: Generated new TSK 0
I20260810 03:44:13.811409 10220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:13.826515 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:38175
--builtin_ntp_servers=127.21.175.84:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:44:13.969638 10231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:13.969960 10231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:13.970005 10231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:13.974321 10231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:13.974397 10231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:13.974457 10231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:44:13.979439 10231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:38175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:13.980787 10231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:13.982301 10231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:13.993028 10238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:13.993052 10240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:13.993104 10237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:13.993371 10231 server_base.cc:1034] running on GCE node
I20260810 03:44:13.993884 10231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:13.994598 10231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:13.995814 10231 hybrid_clock.cc:648] HybridClock initialized: now 1786333453995793 us; error 34 us; skew 500 ppm
I20260810 03:44:14.003268 10231 webserver.cc:460] Webserver started at http://127.21.175.65:33601/ using document root <none> and password file <none>
I20260810 03:44:14.004132 10231 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:14.004244 10231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:14.004535 10231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:14.007390 10231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "d8fc59a04cd44ad7ac98285f07249ab1"
format_stamp: "Formatted at 2026-08-10 03:44:13 on dist-test-slave-ss4n"
server_key: "932b0993492f9dc83a9b4ba770d00623"
server_key_iv: "dabf17b0d16a9a7e91d2ee07dbed60fb"
server_key_version: "encryptionkey@0"
I20260810 03:44:14.008020 10231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "d8fc59a04cd44ad7ac98285f07249ab1"
format_stamp: "Formatted at 2026-08-10 03:44:13 on dist-test-slave-ss4n"
server_key: "932b0993492f9dc83a9b4ba770d00623"
server_key_iv: "dabf17b0d16a9a7e91d2ee07dbed60fb"
server_key_version: "encryptionkey@0"
I20260810 03:44:14.012327 10231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:44:14.015568 10246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:14.017206 10231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:44:14.017371 10231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "d8fc59a04cd44ad7ac98285f07249ab1"
format_stamp: "Formatted at 2026-08-10 03:44:13 on dist-test-slave-ss4n"
server_key: "932b0993492f9dc83a9b4ba770d00623"
server_key_iv: "dabf17b0d16a9a7e91d2ee07dbed60fb"
server_key_version: "encryptionkey@0"
I20260810 03:44:14.017549 10231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:14.032720 10231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:14.033602 10231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:14.033821 10231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:14.034503 10231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:14.035598 10231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:14.035679 10231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:14.035763 10231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:14.035822 10231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:14.047344 10231 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:42109
I20260810 03:44:14.047376 10374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:42109 every 8 connection(s)
I20260810 03:44:14.048532 10231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:44:14.057129 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 10231
I20260810 03:44:14.057274 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:44:14.057762 22205 external_mini_cluster.cc:1428] Setting key b90123b96305b7e210b1618d5afa2c09
I20260810 03:44:14.059438 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.050815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:50933 (local address 127.21.175.126:38175)
0810 03:44:14.050949 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:14.050954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:14.052186 (+ 1232us) server_negotiation.cc:407] Connection header received
0810 03:44:14.053725 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:14.053728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:14.053779 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:14.053862 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:14.054855 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.055386 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.056677 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.057034 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.057171 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.058828 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:14.058831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:14.058833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:14.058845 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:14.058864 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:14.058871 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:14.058933 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:14.059180 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 03:44:14.059270 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:44:14.060065 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:38175
--builtin_ntp_servers=127.21.175.84:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:44:14.060529 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.051248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38175 (local address 127.21.175.65:50933)
0810 03:44:14.052024 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:14.052072 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:14.053465 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:14.054014 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:14.054021 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:14.054115 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:14.054688 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:14.054703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.055511 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.055514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:14.056532 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:14.056543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.057202 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.057240 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:14.057615 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.058494 (+ 879us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:44:14.058496 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:14.058580 (+ 84us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:44:14.058675 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:14.058963 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:14.058974 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:14.058981 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:44:14.059130 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 03:44:14.059509 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":140,"threads_started":1}
I20260810 03:44:14.062206 10375 heartbeater.cc:347] Connected to a master server at 127.21.175.126:38175
I20260810 03:44:14.062492 10375 heartbeater.cc:464] Registering TS with master...
I20260810 03:44:14.063107 10375 heartbeater.cc:511] Master 127.21.175.126:38175 requested a full tablet report, sending...
I20260810 03:44:14.064747 10162 ts_manager.cc:194] Registered new tserver with Master: d8fc59a04cd44ad7ac98285f07249ab1 (127.21.175.65:42109)
I20260810 03:44:14.066231 10162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.175.65:50933
W20260810 03:44:14.176976 10378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:14.177297 10378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:14.177372 10378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:14.181455 10378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:14.181584 10378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:14.181682 10378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:44:14.186586 10378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:38175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:14.187752 10378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:14.189287 10378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:14.199391 10384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:14.199391 10386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:14.199414 10383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:14.199805 10378 server_base.cc:1034] running on GCE node
I20260810 03:44:14.200181 10378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:14.200825 10378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:14.201994 10378 hybrid_clock.cc:648] HybridClock initialized: now 1786333454201978 us; error 27 us; skew 500 ppm
I20260810 03:44:14.204145 10378 webserver.cc:460] Webserver started at http://127.21.175.66:33889/ using document root <none> and password file <none>
I20260810 03:44:14.204684 10378 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:14.204744 10378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:14.204905 10378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:14.206961 10378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "0372a981540f4d68b2f2a3ba6ce5fbfb"
format_stamp: "Formatted at 2026-08-10 03:44:14 on dist-test-slave-ss4n"
server_key: "8599accc6f698f83d5cc4c498d82b677"
server_key_iv: "6a0ac372e87f0b78f19132141639d7fb"
server_key_version: "encryptionkey@0"
I20260810 03:44:14.207543 10378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "0372a981540f4d68b2f2a3ba6ce5fbfb"
format_stamp: "Formatted at 2026-08-10 03:44:14 on dist-test-slave-ss4n"
server_key: "8599accc6f698f83d5cc4c498d82b677"
server_key_iv: "6a0ac372e87f0b78f19132141639d7fb"
server_key_version: "encryptionkey@0"
I20260810 03:44:14.211381 10378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:44:14.213935 10392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:14.215216 10378 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:44:14.215319 10378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "0372a981540f4d68b2f2a3ba6ce5fbfb"
format_stamp: "Formatted at 2026-08-10 03:44:14 on dist-test-slave-ss4n"
server_key: "8599accc6f698f83d5cc4c498d82b677"
server_key_iv: "6a0ac372e87f0b78f19132141639d7fb"
server_key_version: "encryptionkey@0"
I20260810 03:44:14.215453 10378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:14.254186 10378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:14.254943 10378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:14.255157 10378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:14.255784 10378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:14.256759 10378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:14.256831 10378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:14.256903 10378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:14.256958 10378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:14.266443 10378 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:33561
I20260810 03:44:14.266489 10526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:33561 every 8 connection(s)
I20260810 03:44:14.267405 10378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:44:14.268548 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 10378
I20260810 03:44:14.268678 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:44:14.268982 22205 external_mini_cluster.cc:1428] Setting key afb386e64543a5a9ffe66663a7a89c5d
I20260810 03:44:14.271153 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:38175
--builtin_ntp_servers=127.21.175.84:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:44:14.277503 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.269626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:52713 (local address 127.21.175.126:38175)
0810 03:44:14.269770 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:14.269774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:14.270913 (+ 1139us) server_negotiation.cc:407] Connection header received
0810 03:44:14.272415 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:14.272418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:14.272475 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:14.272556 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:14.273692 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.274229 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.275409 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.275654 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.275770 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.276960 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:14.276964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:14.276967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:14.276980 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:14.277010 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:14.277018 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:14.277103 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:14.277270 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 03:44:14.277336 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:44:14.278461 10530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.270009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38175 (local address 127.21.175.66:52713)
0810 03:44:14.270771 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:14.270813 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:14.272197 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:14.272852 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:14.272861 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:14.272977 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:14.273531 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:14.273560 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.274364 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.274367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:14.275279 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:14.275288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.275818 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.275822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:14.275988 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.276692 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:44:14.276693 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:14.276735 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:44:14.276800 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:14.277111 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:14.277122 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:14.277130 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 03:44:14.277258 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 03:44:14.277603 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":145,"threads_started":1}
I20260810 03:44:14.279735 10527 heartbeater.cc:347] Connected to a master server at 127.21.175.126:38175
I20260810 03:44:14.279989 10527 heartbeater.cc:464] Registering TS with master...
I20260810 03:44:14.280525 10527 heartbeater.cc:511] Master 127.21.175.126:38175 requested a full tablet report, sending...
I20260810 03:44:14.281714 10162 ts_manager.cc:194] Registered new tserver with Master: 0372a981540f4d68b2f2a3ba6ce5fbfb (127.21.175.66:33561)
I20260810 03:44:14.282634 10162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.175.66:52713
W20260810 03:44:14.388653 10531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:14.389056 10531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:14.389103 10531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:14.393205 10531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:14.393275 10531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:14.393332 10531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:44:14.398595 10531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:38175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:14.399649 10531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:14.400988 10531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:14.411304 10537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:14.411284 10538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:14.411293 10531 server_base.cc:1034] running on GCE node
W20260810 03:44:14.411325 10541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:14.411978 10531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:14.412678 10531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:14.413861 10531 hybrid_clock.cc:648] HybridClock initialized: now 1786333454413840 us; error 32 us; skew 500 ppm
I20260810 03:44:14.415827 10531 webserver.cc:460] Webserver started at http://127.21.175.67:42323/ using document root <none> and password file <none>
I20260810 03:44:14.416424 10531 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:14.416531 10531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:14.416774 10531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:14.419014 10531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "ff19a0c6f72d49019913505bef5780d3"
format_stamp: "Formatted at 2026-08-10 03:44:14 on dist-test-slave-ss4n"
server_key: "1e969514fdd03b9691f580c61ab7b2bd"
server_key_iv: "fd2977509c4f2c04cc27a184d5d91203"
server_key_version: "encryptionkey@0"
I20260810 03:44:14.419601 10531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "ff19a0c6f72d49019913505bef5780d3"
format_stamp: "Formatted at 2026-08-10 03:44:14 on dist-test-slave-ss4n"
server_key: "1e969514fdd03b9691f580c61ab7b2bd"
server_key_iv: "fd2977509c4f2c04cc27a184d5d91203"
server_key_version: "encryptionkey@0"
I20260810 03:44:14.423568 10531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:44:14.426222 10547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:14.427588 10531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:44:14.427742 10531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "ff19a0c6f72d49019913505bef5780d3"
format_stamp: "Formatted at 2026-08-10 03:44:14 on dist-test-slave-ss4n"
server_key: "1e969514fdd03b9691f580c61ab7b2bd"
server_key_iv: "fd2977509c4f2c04cc27a184d5d91203"
server_key_version: "encryptionkey@0"
I20260810 03:44:14.427870 10531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:14.444480 10531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:14.445240 10531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:14.445452 10531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:14.446122 10531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:14.447116 10531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:14.447187 10531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:14.447263 10531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:14.447299 10531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:14.456614 10531 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:36061
I20260810 03:44:14.456636 10680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:36061 every 8 connection(s)
I20260810 03:44:14.457664 10531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:44:14.459009 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 10531
I20260810 03:44:14.459152 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:44:14.459461 22205 external_mini_cluster.cc:1428] Setting key 34bcbf3ed7fa11bcbbdfaaec309d9897
I20260810 03:44:14.467761 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.460634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:57909 (local address 127.21.175.126:38175)
0810 03:44:14.460762 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:14.460767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:14.461294 (+ 527us) server_negotiation.cc:407] Connection header received
0810 03:44:14.462902 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:14.462905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:14.462964 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:14.463050 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:14.464007 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.464546 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.465797 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.466023 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.466144 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.467255 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:14.467259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:14.467261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:14.467281 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:14.467301 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:14.467309 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:14.467384 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:14.467549 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 03:44:14.467614 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:44:14.468684 10683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.460313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:38175 (local address 127.21.175.67:57909)
0810 03:44:14.461156 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:14.461200 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:14.462685 (+ 1485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:14.463257 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:14.463265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:14.463355 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:14.463862 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:14.463875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.464689 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.464693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:14.465670 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:14.465681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.466182 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.466187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:14.466359 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.466989 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:44:14.466991 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:14.467033 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:44:14.467100 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:14.467399 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:14.467416 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:14.467422 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 03:44:14.467549 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 03:44:14.467915 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":178,"threads_started":1}
I20260810 03:44:14.469972 10681 heartbeater.cc:347] Connected to a master server at 127.21.175.126:38175
I20260810 03:44:14.470201 10681 heartbeater.cc:464] Registering TS with master...
I20260810 03:44:14.470706 10681 heartbeater.cc:511] Master 127.21.175.126:38175 requested a full tablet report, sending...
I20260810 03:44:14.471789 10162 ts_manager.cc:194] Registered new tserver with Master: ff19a0c6f72d49019913505bef5780d3 (127.21.175.67:36061)
I20260810 03:44:14.472332 10162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.175.67:57909
I20260810 03:44:14.473927 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:44:14.480425 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.476237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43326 (local address 127.21.175.126:38175)
0810 03:44:14.476364 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:14.476367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:14.476535 (+ 168us) server_negotiation.cc:407] Connection header received
0810 03:44:14.476618 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:14.476620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:14.476661 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:14.476736 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:14.477057 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.477659 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.478630 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.478868 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.478970 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.479958 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:14.479961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:14.479963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:14.479975 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:14.479995 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:14.480003 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:14.480132 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:14.480273 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 03:44:14.480310 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 03:44:14.483865 10162 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:44:14.486927 10162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:44:14.501754 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.495606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:33561 (local address 127.0.0.1:50356)
0810 03:44:14.496494 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:14.496509 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:14.496612 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:14.497644 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:14.497647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:14.497676 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:14.497781 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:14.497791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.499070 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.499073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:14.500240 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:14.500247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.501261 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.501265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:14.501348 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.501371 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 03:44:14.501422 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 03:44:14.501553 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":88,"spinlock_wait_cycles":768,"thread_start_us":104,"threads_started":1}
I20260810 03:44:14.502107 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.495463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:42109 (local address 127.0.0.1:54400)
0810 03:44:14.495937 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:14.495984 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:14.496116 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:14.496965 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:14.496971 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:14.496994 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:14.497166 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:14.497176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.498686 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.498690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:14.500220 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:14.500232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.501106 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.501110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:14.501218 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.501255 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 03:44:14.501379 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 03:44:14.501958 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":55,"thread_start_us":97,"threads_started":1}
I20260810 03:44:14.502177 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.495624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.21.175.65:42109)
0810 03:44:14.496036 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:14.496053 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:14.496081 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:44:14.496338 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:14.496346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:14.496608 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:14.496772 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:14.497300 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.498508 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.500369 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.500965 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.501097 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.501304 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:14.501410 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 03:44:14.501779 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":99,"threads_started":1}
I20260810 03:44:14.502341 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.496675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.21.175.67:36061)
0810 03:44:14.497129 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:14.497143 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:14.497166 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:44:14.497244 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:14.497249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:14.497449 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:14.497645 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:14.498018 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.498864 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.500317 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.501119 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.501237 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.501420 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:14.501825 (+ 405us) server_negotiation.cc:299] Negotiation successful
0810 03:44:14.502035 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":106,"threads_started":1}
I20260810 03:44:14.502386 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.496308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50356 (local address 127.21.175.66:33561)
0810 03:44:14.496753 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:14.496764 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:14.496786 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:44:14.497226 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:14.497236 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:14.497452 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:14.497744 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:14.497873 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.498974 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.500378 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.501118 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.501551 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.501730 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:14.501781 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 03:44:14.502070 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":139,"threads_started":1}
I20260810 03:44:14.502436 10697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.496261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:36061 (local address 127.0.0.1:38972)
0810 03:44:14.496985 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:14.497000 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:14.497089 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:14.497621 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:14.497624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:14.497643 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:14.497916 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:14.497920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.499010 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.499016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:14.500195 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:14.500205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.501518 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.501521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:14.501595 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.501615 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 03:44:14.501684 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 03:44:14.501752 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":68,"spinlock_wait_cycles":2304,"thread_start_us":93,"threads_started":1}
I20260810 03:44:14.505074 10303 tablet_service.cc:1467] Processing CreateTablet for tablet edecf8b06ace40e6893176f670f3bb92 (DEFAULT_TABLE table=test-table [id=7ed67efda0f1422ab3db763e2e46be42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:14.505523 10606 tablet_service.cc:1467] Processing CreateTablet for tablet edecf8b06ace40e6893176f670f3bb92 (DEFAULT_TABLE table=test-table [id=7ed67efda0f1422ab3db763e2e46be42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:14.505515 10450 tablet_service.cc:1467] Processing CreateTablet for tablet edecf8b06ace40e6893176f670f3bb92 (DEFAULT_TABLE table=test-table [id=7ed67efda0f1422ab3db763e2e46be42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:14.506115 10303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edecf8b06ace40e6893176f670f3bb92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:14.506531 10450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edecf8b06ace40e6893176f670f3bb92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:14.506529 10606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edecf8b06ace40e6893176f670f3bb92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:14.512521 10700 tablet_bootstrap.cc:492] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1: Bootstrap starting.
I20260810 03:44:14.514497 10701 tablet_bootstrap.cc:492] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb: Bootstrap starting.
I20260810 03:44:14.515246 10700 tablet_bootstrap.cc:654] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:14.516386 10700 log.cc:824] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:14.516604 10701 tablet_bootstrap.cc:654] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:14.517081 10702 tablet_bootstrap.cc:492] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3: Bootstrap starting.
I20260810 03:44:14.517465 10701 log.cc:824] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:14.518585 10700 tablet_bootstrap.cc:492] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1: No bootstrap required, opened a new log
I20260810 03:44:14.518781 10700 ts_tablet_manager.cc:1397] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:44:14.519243 10702 tablet_bootstrap.cc:654] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:14.519704 10701 tablet_bootstrap.cc:492] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb: No bootstrap required, opened a new log
I20260810 03:44:14.519881 10701 ts_tablet_manager.cc:1397] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 03:44:14.520123 10702 log.cc:824] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:14.522784 10702 tablet_bootstrap.cc:492] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3: No bootstrap required, opened a new log
I20260810 03:44:14.522981 10702 ts_tablet_manager.cc:1397] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:44:14.523295 10700 raft_consensus.cc:348] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc59a04cd44ad7ac98285f07249ab1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42109 } } peers { permanent_uuid: "0372a981540f4d68b2f2a3ba6ce5fbfb" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 33561 } } peers { permanent_uuid: "ff19a0c6f72d49019913505bef5780d3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36061 } }
I20260810 03:44:14.523741 10700 raft_consensus.cc:374] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:14.523806 10700 raft_consensus.cc:729] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8fc59a04cd44ad7ac98285f07249ab1, State: Initialized, Role: FOLLOWER
I20260810 03:44:14.524143 10701 raft_consensus.cc:348] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc59a04cd44ad7ac98285f07249ab1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42109 } } peers { permanent_uuid: "0372a981540f4d68b2f2a3ba6ce5fbfb" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 33561 } } peers { permanent_uuid: "ff19a0c6f72d49019913505bef5780d3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36061 } }
I20260810 03:44:14.524263 10700 consensus_queue.cc:260] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc59a04cd44ad7ac98285f07249ab1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42109 } } peers { permanent_uuid: "0372a981540f4d68b2f2a3ba6ce5fbfb" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 33561 } } peers { permanent_uuid: "ff19a0c6f72d49019913505bef5780d3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36061 } }
I20260810 03:44:14.524396 10701 raft_consensus.cc:374] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:14.524595 10701 raft_consensus.cc:729] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0372a981540f4d68b2f2a3ba6ce5fbfb, State: Initialized, Role: FOLLOWER
I20260810 03:44:14.525102 10701 consensus_queue.cc:260] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc59a04cd44ad7ac98285f07249ab1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42109 } } peers { permanent_uuid: "0372a981540f4d68b2f2a3ba6ce5fbfb" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 33561 } } peers { permanent_uuid: "ff19a0c6f72d49019913505bef5780d3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36061 } }
I20260810 03:44:14.525583 10700 ts_tablet_manager.cc:1428] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 03:44:14.525982 10527 heartbeater.cc:503] Master 127.21.175.126:38175 was elected leader, sending a full tablet report...
I20260810 03:44:14.526016 10375 heartbeater.cc:503] Master 127.21.175.126:38175 was elected leader, sending a full tablet report...
I20260810 03:44:14.527172 10701 ts_tablet_manager.cc:1428] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260810 03:44:14.527060 10702 raft_consensus.cc:348] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc59a04cd44ad7ac98285f07249ab1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42109 } } peers { permanent_uuid: "0372a981540f4d68b2f2a3ba6ce5fbfb" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 33561 } } peers { permanent_uuid: "ff19a0c6f72d49019913505bef5780d3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36061 } }
I20260810 03:44:14.527308 10702 raft_consensus.cc:374] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:14.527356 10702 raft_consensus.cc:729] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff19a0c6f72d49019913505bef5780d3, State: Initialized, Role: FOLLOWER
I20260810 03:44:14.527769 10702 consensus_queue.cc:260] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc59a04cd44ad7ac98285f07249ab1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42109 } } peers { permanent_uuid: "0372a981540f4d68b2f2a3ba6ce5fbfb" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 33561 } } peers { permanent_uuid: "ff19a0c6f72d49019913505bef5780d3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36061 } }
I20260810 03:44:14.528769 10702 ts_tablet_manager.cc:1428] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260810 03:44:14.529102 10681 heartbeater.cc:503] Master 127.21.175.126:38175 was elected leader, sending a full tablet report...
W20260810 03:44:14.549858 10376 tablet.cc:2369] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:44:14.558396 10706 raft_consensus.cc:482] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:44:14.558648 10706 raft_consensus.cc:504] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc59a04cd44ad7ac98285f07249ab1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42109 } } peers { permanent_uuid: "0372a981540f4d68b2f2a3ba6ce5fbfb" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 33561 } } peers { permanent_uuid: "ff19a0c6f72d49019913505bef5780d3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36061 } }
I20260810 03:44:14.559942 10706 leader_election.cc:288] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0372a981540f4d68b2f2a3ba6ce5fbfb (127.21.175.66:33561), ff19a0c6f72d49019913505bef5780d3 (127.21.175.67:36061)
I20260810 03:44:14.563925 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.560417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:53609 (local address 127.21.175.66:33561)
0810 03:44:14.560557 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:14.560562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:14.560580 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:44:14.560666 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:14.560671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:14.560730 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:14.560815 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:14.561234 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.561951 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.562997 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.563373 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.563504 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.563558 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:14.563741 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 03:44:14.563811 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:44:14.563925 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.560217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:33561 (local address 127.21.175.65:53609)
0810 03:44:14.560381 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:14.560399 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:14.560562 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:14.560938 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:14.560941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:14.560964 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:14.561097 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:14.561104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.562067 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.562072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:14.562861 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:14.562870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.563494 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.563498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:14.563600 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.563629 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 03:44:14.563722 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 03:44:14.563795 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 03:44:14.564670 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.560889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:55775 (local address 127.21.175.67:36061)
0810 03:44:14.561067 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:14.561073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:14.561090 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:44:14.561155 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:14.561159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:14.561211 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:14.561312 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:14.561620 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.562263 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.563514 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.563962 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.564128 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.564206 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:14.564489 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 03:44:14.564559 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 03:44:14.564687 10709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.560510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:36061 (local address 127.21.175.65:55775)
0810 03:44:14.560941 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:14.560954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:14.561027 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:14.561341 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:14.561343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:14.561363 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:14.561504 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:14.561512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.562415 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.562419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:14.563352 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:14.563363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.564251 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.564255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:14.564355 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.564374 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 03:44:14.564470 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 03:44:14.564539 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":152,"threads_started":1}
I20260810 03:44:14.564672 10474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edecf8b06ace40e6893176f670f3bb92" candidate_uuid: "d8fc59a04cd44ad7ac98285f07249ab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0372a981540f4d68b2f2a3ba6ce5fbfb" is_pre_election: true
I20260810 03:44:14.565106 10474 raft_consensus.cc:2454] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8fc59a04cd44ad7ac98285f07249ab1 in term 0.
I20260810 03:44:14.565312 10628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edecf8b06ace40e6893176f670f3bb92" candidate_uuid: "d8fc59a04cd44ad7ac98285f07249ab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff19a0c6f72d49019913505bef5780d3" is_pre_election: true
I20260810 03:44:14.565688 10628 raft_consensus.cc:2454] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8fc59a04cd44ad7ac98285f07249ab1 in term 0.
I20260810 03:44:14.565712 10248 leader_election.cc:302] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0372a981540f4d68b2f2a3ba6ce5fbfb, d8fc59a04cd44ad7ac98285f07249ab1; no voters:
I20260810 03:44:14.566056 10706 raft_consensus.cc:2781] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:44:14.566166 10706 raft_consensus.cc:482] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:44:14.566226 10706 raft_consensus.cc:3037] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:14.567420 10706 raft_consensus.cc:504] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc59a04cd44ad7ac98285f07249ab1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42109 } } peers { permanent_uuid: "0372a981540f4d68b2f2a3ba6ce5fbfb" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 33561 } } peers { permanent_uuid: "ff19a0c6f72d49019913505bef5780d3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36061 } }
I20260810 03:44:14.567804 10706 leader_election.cc:288] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [CANDIDATE]: Term 1 election: Requested vote from peers 0372a981540f4d68b2f2a3ba6ce5fbfb (127.21.175.66:33561), ff19a0c6f72d49019913505bef5780d3 (127.21.175.67:36061)
I20260810 03:44:14.568162 10474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edecf8b06ace40e6893176f670f3bb92" candidate_uuid: "d8fc59a04cd44ad7ac98285f07249ab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0372a981540f4d68b2f2a3ba6ce5fbfb"
I20260810 03:44:14.568316 10628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edecf8b06ace40e6893176f670f3bb92" candidate_uuid: "d8fc59a04cd44ad7ac98285f07249ab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff19a0c6f72d49019913505bef5780d3"
I20260810 03:44:14.568310 10474 raft_consensus.cc:3037] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:14.568485 10628 raft_consensus.cc:3037] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:14.569468 10474 raft_consensus.cc:2454] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8fc59a04cd44ad7ac98285f07249ab1 in term 1.
I20260810 03:44:14.569778 10628 raft_consensus.cc:2454] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8fc59a04cd44ad7ac98285f07249ab1 in term 1.
I20260810 03:44:14.569916 10248 leader_election.cc:302] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0372a981540f4d68b2f2a3ba6ce5fbfb, d8fc59a04cd44ad7ac98285f07249ab1; no voters:
I20260810 03:44:14.570147 10706 raft_consensus.cc:2781] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:14.570410 10706 raft_consensus.cc:686] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [term 1 LEADER]: Becoming Leader. State: Replica: d8fc59a04cd44ad7ac98285f07249ab1, State: Running, Role: LEADER
I20260810 03:44:14.570781 10706 consensus_queue.cc:237] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc59a04cd44ad7ac98285f07249ab1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42109 } } peers { permanent_uuid: "0372a981540f4d68b2f2a3ba6ce5fbfb" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 33561 } } peers { permanent_uuid: "ff19a0c6f72d49019913505bef5780d3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36061 } }
I20260810 03:44:14.574215 10160 catalog_manager.cc:5495] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8fc59a04cd44ad7ac98285f07249ab1 (127.21.175.65). New cstate: current_term: 1 leader_uuid: "d8fc59a04cd44ad7ac98285f07249ab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc59a04cd44ad7ac98285f07249ab1" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 42109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0372a981540f4d68b2f2a3ba6ce5fbfb" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 33561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff19a0c6f72d49019913505bef5780d3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36061 } health_report { overall_health: UNKNOWN } } }
I20260810 03:44:14.586395 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.582543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.21.175.65:42109)
0810 03:44:14.582692 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:14.582695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:14.582709 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:44:14.582817 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:14.582819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:14.582873 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:14.582963 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:44:14.583292 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.583795 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.584702 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:14.584904 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:14.584982 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:14.585381 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:44:14.585891 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:44:14.586010 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:14.586162 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 03:44:14.586226 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:44:14.592574 10628 raft_consensus.cc:1261] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3 [term 1 FOLLOWER]: Refusing update from remote peer d8fc59a04cd44ad7ac98285f07249ab1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:44:14.592574 10474 raft_consensus.cc:1261] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb [term 1 FOLLOWER]: Refusing update from remote peer d8fc59a04cd44ad7ac98285f07249ab1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:44:14.593325 10706 consensus_queue.cc:1035] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff19a0c6f72d49019913505bef5780d3" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:44:14.593611 10710 consensus_queue.cc:1035] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0372a981540f4d68b2f2a3ba6ce5fbfb" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 33561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:44:14.601298 10714 mvcc.cc:205] Tried to move back new op lower bound from 7316821830003945472 to 7316821829924573184. Current Snapshot: MvccSnapshot[applied={T|T < 7316821830003945472}]
I20260810 03:44:14.604694 10715 mvcc.cc:205] Tried to move back new op lower bound from 7316821830003945472 to 7316821829924573184. Current Snapshot: MvccSnapshot[applied={T|T < 7316821830003945472}]
I20260810 03:44:14.611915 10160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:44:14.612116 10160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:44:14.614830 10160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c23945d1cd8463a9c1562f7b9293cb7: Sending DeleteTablet for 3 replicas of tablet edecf8b06ace40e6893176f670f3bb92
I20260810 03:44:14.615653 10303 tablet_service.cc:1514] Processing DeleteTablet for tablet edecf8b06ace40e6893176f670f3bb92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:14 UTC) from {username='slave'} at 127.0.0.1:54400
I20260810 03:44:14.615772 10450 tablet_service.cc:1514] Processing DeleteTablet for tablet edecf8b06ace40e6893176f670f3bb92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:14 UTC) from {username='slave'} at 127.0.0.1:50356
I20260810 03:44:14.615984 10606 tablet_service.cc:1514] Processing DeleteTablet for tablet edecf8b06ace40e6893176f670f3bb92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:14 UTC) from {username='slave'} at 127.0.0.1:38972
I20260810 03:44:14.616448 10725 tablet_replica.cc:330] T edecf8b06ace40e6893176f670f3bb92 P d8fc59a04cd44ad7ac98285f07249ab1: stopping tablet replica
I20260810 03:44:14.616501 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 10231
I20260810 03:44:14.617763 10726 tablet_replica.cc:330] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb: stopping tablet replica
I20260810 03:44:14.618105 10726 raft_consensus.cc:2229] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:44:14.618439 10726 raft_consensus.cc:2258] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:44:14.618969 10727 tablet_replica.cc:330] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3: stopping tablet replica
I20260810 03:44:14.619269 10727 raft_consensus.cc:2229] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:44:14.619530 10727 raft_consensus.cc:2258] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:44:14.622176 10727 ts_tablet_manager.cc:1905] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:44:14.623374 10726 ts_tablet_manager.cc:1905] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:44:14.624717 10727 ts_tablet_manager.cc:1918] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:44:14.624852 10727 log.cc:1196] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/edecf8b06ace40e6893176f670f3bb92
I20260810 03:44:14.625206 10727 ts_tablet_manager.cc:1939] T edecf8b06ace40e6893176f670f3bb92 P ff19a0c6f72d49019913505bef5780d3: Deleting consensus metadata
I20260810 03:44:14.626714 10147 catalog_manager.cc:4841] TS ff19a0c6f72d49019913505bef5780d3 (127.21.175.67:36061): tablet edecf8b06ace40e6893176f670f3bb92 (table test-table [id=7ed67efda0f1422ab3db763e2e46be42]) successfully deleted
I20260810 03:44:14.626896 10726 ts_tablet_manager.cc:1918] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:44:14.627019 10726 log.cc:1196] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/edecf8b06ace40e6893176f670f3bb92
I20260810 03:44:14.627403 10726 ts_tablet_manager.cc:1939] T edecf8b06ace40e6893176f670f3bb92 P 0372a981540f4d68b2f2a3ba6ce5fbfb: Deleting consensus metadata
W20260810 03:44:14.628417 10146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.175.65:42109 (error 108)
I20260810 03:44:14.628623 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 10378
I20260810 03:44:14.628726 10147 catalog_manager.cc:4841] TS 0372a981540f4d68b2f2a3ba6ce5fbfb (127.21.175.66:33561): tablet edecf8b06ace40e6893176f670f3bb92 (table test-table [id=7ed67efda0f1422ab3db763e2e46be42]) successfully deleted
I20260810 03:44:14.632124 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:14.631719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:42109 (local address 127.0.0.1:54416)
0810 03:44:14.631940 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:14.632018 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:42109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260810 03:44:14.632421 10146 catalog_manager.cc:4584] TS d8fc59a04cd44ad7ac98285f07249ab1 (127.21.175.65:42109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet edecf8b06ace40e6893176f670f3bb92: Network error: Client connection negotiation failed: client connection to 127.21.175.65:42109: connect: Connection refused (error 111)
I20260810 03:44:14.638134 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 10531
I20260810 03:44:14.646512 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 10128
2026-08-10T03:44:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1144 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:14 dist-test-slave-ss4n krb5kdc[10731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:14 dist-test-slave-ss4n krb5kdc[10731](info): set up 2 sockets
Aug 10 03:44:14 dist-test-slave-ss4n krb5kdc[10731](info): commencing operation
krb5kdc: starting...
W20260810 03:44:15.745976 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:15 dist-test-slave-ss4n krb5kdc[10731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:44:15.886236 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:37933
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:37933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:16.004557 10747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:16.004878 10747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:16.004966 10747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:16.009171 10747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:16.009269 10747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:16.009307 10747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:16.009346 10747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:16.009413 10747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:16.016510 10747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:37933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:37933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:16.017748 10747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:16.018775 10747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:16.027568 10754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:16.027805 10753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:16.027552 10756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:16.027786 10747 server_base.cc:1034] running on GCE node
I20260810 03:44:16.028484 10747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:16.029640 10747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:16.030826 10747 hybrid_clock.cc:648] HybridClock initialized: now 1786333456030794 us; error 48 us; skew 500 ppm
Aug 10 03:44:16 dist-test-slave-ss4n krb5kdc[10731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333456, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:16.033779 10747 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:44:16.034766 10747 webserver.cc:460] Webserver started at http://127.21.175.126:45781/ using document root <none> and password file <none>
I20260810 03:44:16.035360 10747 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:16.035458 10747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:16.035704 10747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:16.037919 10747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "1c466747caaa442ea49d569f7a9e5396"
format_stamp: "Formatted at 2026-08-10 03:44:16 on dist-test-slave-ss4n"
server_key: "dbaf7bbc115195d3babb44798e2e294a"
server_key_iv: "7b3df6867dbdff047d5079c4ba0eeedc"
server_key_version: "encryptionkey@0"
I20260810 03:44:16.038551 10747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "1c466747caaa442ea49d569f7a9e5396"
format_stamp: "Formatted at 2026-08-10 03:44:16 on dist-test-slave-ss4n"
server_key: "dbaf7bbc115195d3babb44798e2e294a"
server_key_iv: "7b3df6867dbdff047d5079c4ba0eeedc"
server_key_version: "encryptionkey@0"
I20260810 03:44:16.042865 10747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:44:16.046010 10764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:16.047609 10747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:44:16.047801 10747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "1c466747caaa442ea49d569f7a9e5396"
format_stamp: "Formatted at 2026-08-10 03:44:16 on dist-test-slave-ss4n"
server_key: "dbaf7bbc115195d3babb44798e2e294a"
server_key_iv: "7b3df6867dbdff047d5079c4ba0eeedc"
server_key_version: "encryptionkey@0"
I20260810 03:44:16.047952 10747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:16.063350 10747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:16.067196 10747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:16.067456 10747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:16.075614 10747 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:37933
I20260810 03:44:16.075647 10825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:37933 every 8 connection(s)
I20260810 03:44:16.076632 10747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:16.079689 10826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:16.085183 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 10747
I20260810 03:44:16.085353 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:16.085737 22205 external_mini_cluster.cc:1428] Setting key f18551963b7bbff990916e53a4040360
I20260810 03:44:16.086199 10826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396: Bootstrap starting.
I20260810 03:44:16.088972 10826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:16.089854 10826 log.cc:824] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:16.091984 10826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396: No bootstrap required, opened a new log
Aug 10 03:44:16 dist-test-slave-ss4n krb5kdc[10731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:16.096422 10826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c466747caaa442ea49d569f7a9e5396" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37933 } }
I20260810 03:44:16.096736 10826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:16.096828 10826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c466747caaa442ea49d569f7a9e5396, State: Initialized, Role: FOLLOWER
I20260810 03:44:16.097400 10826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c466747caaa442ea49d569f7a9e5396" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37933 } }
I20260810 03:44:16.097610 10826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:16.097699 10826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:16.097851 10826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:16.099035 10826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c466747caaa442ea49d569f7a9e5396" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37933 } }
I20260810 03:44:16.099460 10826 leader_election.cc:302] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c466747caaa442ea49d569f7a9e5396; no voters:
I20260810 03:44:16.099879 10826 leader_election.cc:288] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:16.099992 10831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:16.100330 10831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [term 1 LEADER]: Becoming Leader. State: Replica: 1c466747caaa442ea49d569f7a9e5396, State: Running, Role: LEADER
I20260810 03:44:16.100692 10831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c466747caaa442ea49d569f7a9e5396" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37933 } }
I20260810 03:44:16.101560 10826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:16.102470 10832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c466747caaa442ea49d569f7a9e5396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c466747caaa442ea49d569f7a9e5396" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37933 } } }
I20260810 03:44:16.102602 10832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:16.103173 10835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c466747caaa442ea49d569f7a9e5396. Latest consensus state: current_term: 1 leader_uuid: "1c466747caaa442ea49d569f7a9e5396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c466747caaa442ea49d569f7a9e5396" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37933 } } }
I20260810 03:44:16.103276 10835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:16.103508 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:16.087426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34308 (local address 127.21.175.126:37933)
0810 03:44:16.087980 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:16.087998 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:16.088040 (+ 42us) server_negotiation.cc:407] Connection header received
0810 03:44:16.088849 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:16.088869 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:16.089296 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:16.089741 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:16.090792 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.091924 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.092983 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.093617 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.093828 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:16.097054 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:16.097060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:16.097076 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:16.097112 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:16.099296 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:16.099876 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:16.099884 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:16.099892 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:16.100101 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:16.100596 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:16.100600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:16.100601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:16.101164 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:16.101362 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:16.101932 (+ 570us) server_negotiation.cc:299] Negotiation successful
0810 03:44:16.102352 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":160,"threads_started":1}
W20260810 03:44:16.107374 10848 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:44:16.107450 10848 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:44:16.107532 10842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:16.108258 10842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:16.114035 10842 catalog_manager.cc:1347] Generated new cluster ID: fb26bde960dd4e288a6032f010eeb3ed
I20260810 03:44:16.114122 10842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:16.128633 10842 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:16.129901 10842 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:16.141655 10842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396: Generated new TSK 0
I20260810 03:44:16.142531 10842 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
Entry for principal kudu/127.21.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab.
I20260810 03:44:16.181365 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:37933
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:16.299820 10855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:16.300103 10855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:16.300149 10855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:16.304632 10855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:16.304699 10855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:16.304759 10855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:44:16.310290 10855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.65.keytab
--principal=kudu/127.21.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:37933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:16.311339 10855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:16.312337 10855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:16.322638 10864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:16.322631 10860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:16.322640 10861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:16.322903 10855 server_base.cc:1034] running on GCE node
I20260810 03:44:16.323453 10855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:16.324128 10855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:16.325380 10855 hybrid_clock.cc:648] HybridClock initialized: now 1786333456325325 us; error 58 us; skew 500 ppm
Aug 10 03:44:16 dist-test-slave-ss4n krb5kdc[10731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333456, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:16.328763 10855 init.cc:377] Logged in from keytab as kudu/127.21.175.65@KRBTEST.COM (short username kudu)
I20260810 03:44:16.329834 10855 webserver.cc:460] Webserver started at http://127.21.175.65:39815/ using document root <none> and password file <none>
I20260810 03:44:16.330456 10855 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:16.330559 10855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:16.330866 10855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:16.333086 10855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "bb09884d77944656afb9043d4f5c95dd"
format_stamp: "Formatted at 2026-08-10 03:44:16 on dist-test-slave-ss4n"
server_key: "ca9b4ed4e34fa3722e70d0bc1eae8d8e"
server_key_iv: "d5f26a92eecb638041084404007f5ebd"
server_key_version: "encryptionkey@0"
I20260810 03:44:16.333758 10855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "bb09884d77944656afb9043d4f5c95dd"
format_stamp: "Formatted at 2026-08-10 03:44:16 on dist-test-slave-ss4n"
server_key: "ca9b4ed4e34fa3722e70d0bc1eae8d8e"
server_key_iv: "d5f26a92eecb638041084404007f5ebd"
server_key_version: "encryptionkey@0"
I20260810 03:44:16.338047 10855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:44:16.340880 10871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:16.342549 10855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:44:16.342670 10855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "bb09884d77944656afb9043d4f5c95dd"
format_stamp: "Formatted at 2026-08-10 03:44:16 on dist-test-slave-ss4n"
server_key: "ca9b4ed4e34fa3722e70d0bc1eae8d8e"
server_key_iv: "d5f26a92eecb638041084404007f5ebd"
server_key_version: "encryptionkey@0"
I20260810 03:44:16.342772 10855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:16.360922 10855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:16.364675 10855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:16.364843 10855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:16.365406 10855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:16.366345 10855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:16.366396 10855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:16.366438 10855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:16.366454 10855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:16.376417 10855 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:37327
I20260810 03:44:16.376509 10999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:37327 every 8 connection(s)
I20260810 03:44:16.377424 10855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:44:16.379058 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 10855
I20260810 03:44:16.379201 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:44:16.379554 22205 external_mini_cluster.cc:1428] Setting key e0b164fec9658958045afa963484a7a4
Aug 10 03:44:16 dist-test-slave-ss4n krb5kdc[10731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333456, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.65@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.66@KRBTEST.COM" created.
I20260810 03:44:16.395241 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:16.379637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:41785 (local address 127.21.175.126:37933)
0810 03:44:16.379794 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:16.379798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:16.380763 (+ 965us) server_negotiation.cc:407] Connection header received
0810 03:44:16.382408 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:16.382412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:16.382478 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:16.382567 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:16.384309 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.384851 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.386970 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.387207 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.387296 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:16.390810 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:16.390813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:16.390816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:16.390842 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:16.392694 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:16.393312 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:16.393316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:16.393318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:16.393413 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:16.393932 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:16.393935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:16.393936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:16.394141 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:16.394295 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:16.394890 (+ 595us) server_negotiation.cc:299] Negotiation successful
0810 03:44:16.395055 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:44:16.396196 11002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:16.380058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:37933 (local address 127.21.175.65:41785)
0810 03:44:16.380598 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:16.380648 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:16.382178 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:16.382754 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:16.382762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:16.383322 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:16.384121 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:16.384146 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.385033 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.385040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:16.386818 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:16.386835 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.387391 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.387399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:16.387677 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:16.388406 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:16.388435 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:16.390588 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:16.392840 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:16.392847 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:16.392867 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:16.393185 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:16.393534 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:16.393541 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:16.393544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:16.393762 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:16.394262 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:16.394270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:16.394600 (+ 330us) client_negotiation.cc:769] Sending connection context
0810 03:44:16.394826 (+ 226us) client_negotiation.cc:240] Negotiation successful
0810 03:44:16.395236 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":126,"threads_started":1}
I20260810 03:44:16.397747 11000 heartbeater.cc:347] Connected to a master server at 127.21.175.126:37933
I20260810 03:44:16.398070 11000 heartbeater.cc:464] Registering TS with master...
I20260810 03:44:16.398712 11000 heartbeater.cc:511] Master 127.21.175.126:37933 requested a full tablet report, sending...
I20260810 03:44:16.400573 10782 ts_manager.cc:194] Registered new tserver with Master: bb09884d77944656afb9043d4f5c95dd (127.21.175.65:37327)
I20260810 03:44:16.402017 10782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.65@KRBTEST.COM'} at 127.21.175.65:41785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
Entry for principal kudu/127.21.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab.
I20260810 03:44:16.413143 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:37933
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:16.535140 11005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:16.535476 11005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:16.535573 11005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:16.539685 11005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:16.539798 11005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:16.539894 11005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:44:16.545553 11005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.66.keytab
--principal=kudu/127.21.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:37933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:16.546819 11005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:16.548022 11005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:16.558437 11011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:16.558517 11005 server_base.cc:1034] running on GCE node
W20260810 03:44:16.558437 11016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:16.558430 11012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:16.559074 11005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:16.559799 11005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:16.561029 11005 hybrid_clock.cc:648] HybridClock initialized: now 1786333456561009 us; error 37 us; skew 500 ppm
Aug 10 03:44:16 dist-test-slave-ss4n krb5kdc[10731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333456, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:16.564337 11005 init.cc:377] Logged in from keytab as kudu/127.21.175.66@KRBTEST.COM (short username kudu)
I20260810 03:44:16.565361 11005 webserver.cc:460] Webserver started at http://127.21.175.66:34919/ using document root <none> and password file <none>
I20260810 03:44:16.566030 11005 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:16.566136 11005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:16.566478 11005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:16.568851 11005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "76d3681121774e048f635cf834fc7d12"
format_stamp: "Formatted at 2026-08-10 03:44:16 on dist-test-slave-ss4n"
server_key: "d16435edf43a4ee84fe8e8cb65524690"
server_key_iv: "b3fd5fc20d7c135127291b279985a667"
server_key_version: "encryptionkey@0"
I20260810 03:44:16.569535 11005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "76d3681121774e048f635cf834fc7d12"
format_stamp: "Formatted at 2026-08-10 03:44:16 on dist-test-slave-ss4n"
server_key: "d16435edf43a4ee84fe8e8cb65524690"
server_key_iv: "b3fd5fc20d7c135127291b279985a667"
server_key_version: "encryptionkey@0"
I20260810 03:44:16.574007 11005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260810 03:44:16.576961 11024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:16.578312 11005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:44:16.578495 11005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "76d3681121774e048f635cf834fc7d12"
format_stamp: "Formatted at 2026-08-10 03:44:16 on dist-test-slave-ss4n"
server_key: "d16435edf43a4ee84fe8e8cb65524690"
server_key_iv: "b3fd5fc20d7c135127291b279985a667"
server_key_version: "encryptionkey@0"
I20260810 03:44:16.578621 11005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:16.604677 11005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:16.608409 11005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:16.608661 11005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:16.609305 11005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:16.610258 11005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:16.610342 11005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:16.610409 11005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:16.610462 11005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:16.619992 11005 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:37331
I20260810 03:44:16.620020 11151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:37331 every 8 connection(s)
I20260810 03:44:16.620934 11005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:44:16.621793 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 11005
I20260810 03:44:16.621951 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:44:16.622283 22205 external_mini_cluster.cc:1428] Setting key fb4e1fc7de1064c265c2c2e14f786cba
WARNING: no policy specified for kudu/127.21.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.67@KRBTEST.COM" created.
Aug 10 03:44:16 dist-test-slave-ss4n krb5kdc[10731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333456, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.66@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:16.641600 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:16.623119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:41811 (local address 127.21.175.126:37933)
0810 03:44:16.623261 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:16.623265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:16.625929 (+ 2664us) server_negotiation.cc:407] Connection header received
0810 03:44:16.627231 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:16.627237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:16.627302 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:16.627403 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:16.628716 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.629654 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.630821 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.631208 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.631357 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:16.637013 (+ 5656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:16.637017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:16.637020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:16.637056 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:16.638923 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:16.639541 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:16.639548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:16.639553 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:16.639654 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:16.640045 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:16.640052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:16.640056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:16.640337 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:16.640459 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:16.641217 (+ 758us) server_negotiation.cc:299] Negotiation successful
0810 03:44:16.641373 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 03:44:16.642565 11154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:16.623550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:37933 (local address 127.21.175.66:41811)
0810 03:44:16.625797 (+ 2247us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:16.625844 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:16.627010 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:16.627566 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:16.627572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:16.627931 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:16.628558 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:16.628575 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.629780 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.629783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:16.630687 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:16.630697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.631418 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.631424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:16.631653 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:16.632320 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:16.632338 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:16.636806 (+ 4468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:16.639100 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:16.639106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:16.639121 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:16.639417 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:16.639772 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:16.639774 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:16.639776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:16.639938 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:16.640507 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:16.640513 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:16.640878 (+ 365us) client_negotiation.cc:769] Sending connection context
0810 03:44:16.641126 (+ 248us) client_negotiation.cc:240] Negotiation successful
0810 03:44:16.641607 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2048,"thread_start_us":127,"threads_started":1}
I20260810 03:44:16.643911 11152 heartbeater.cc:347] Connected to a master server at 127.21.175.126:37933
I20260810 03:44:16.644161 11152 heartbeater.cc:464] Registering TS with master...
I20260810 03:44:16.644697 11152 heartbeater.cc:511] Master 127.21.175.126:37933 requested a full tablet report, sending...
I20260810 03:44:16.645807 10782 ts_manager.cc:194] Registered new tserver with Master: 76d3681121774e048f635cf834fc7d12 (127.21.175.66:37331)
I20260810 03:44:16.646467 10782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.66@KRBTEST.COM'} at 127.21.175.66:41811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
Entry for principal kudu/127.21.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab.
I20260810 03:44:16.657111 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:37933
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:43021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:16.777734 11159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:16.778013 11159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:16.778057 11159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:16.782198 11159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:16.782261 11159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:16.782320 11159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:44:16.787817 11159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:43021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.67.keytab
--principal=kudu/127.21.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:37933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:16.788892 11159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:16.789887 11159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:16.800664 11166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:16.800668 11164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:16.800720 11168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:16.801064 11159 server_base.cc:1034] running on GCE node
I20260810 03:44:16.801679 11159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:16.802536 11159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:16.803748 11159 hybrid_clock.cc:648] HybridClock initialized: now 1786333456803707 us; error 56 us; skew 500 ppm
Aug 10 03:44:16 dist-test-slave-ss4n krb5kdc[10731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333456, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:16.807054 11159 init.cc:377] Logged in from keytab as kudu/127.21.175.67@KRBTEST.COM (short username kudu)
I20260810 03:44:16.808063 11159 webserver.cc:460] Webserver started at http://127.21.175.67:36959/ using document root <none> and password file <none>
I20260810 03:44:16.808768 11159 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:16.808882 11159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:16.809123 11159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:16.811537 11159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "dcaa1a2425d1417eaba15e330b6dcf1f"
format_stamp: "Formatted at 2026-08-10 03:44:16 on dist-test-slave-ss4n"
server_key: "bdc9084c5127bb30ed19fe6b555e234c"
server_key_iv: "b50f5166fa26686ba4fffcdee4da2e00"
server_key_version: "encryptionkey@0"
I20260810 03:44:16.812219 11159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "dcaa1a2425d1417eaba15e330b6dcf1f"
format_stamp: "Formatted at 2026-08-10 03:44:16 on dist-test-slave-ss4n"
server_key: "bdc9084c5127bb30ed19fe6b555e234c"
server_key_iv: "b50f5166fa26686ba4fffcdee4da2e00"
server_key_version: "encryptionkey@0"
I20260810 03:44:16.816708 11159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:44:16.819634 11177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:16.821201 11159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 03:44:16.821369 11159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "dcaa1a2425d1417eaba15e330b6dcf1f"
format_stamp: "Formatted at 2026-08-10 03:44:16 on dist-test-slave-ss4n"
server_key: "bdc9084c5127bb30ed19fe6b555e234c"
server_key_iv: "b50f5166fa26686ba4fffcdee4da2e00"
server_key_version: "encryptionkey@0"
I20260810 03:44:16.821527 11159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:16.848711 11159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:16.852679 11159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:16.852905 11159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:16.853660 11159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:16.854591 11159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:16.854671 11159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:16.854755 11159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:16.854804 11159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:16.865165 11159 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:39089
I20260810 03:44:16.865192 11298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:39089 every 8 connection(s)
I20260810 03:44:16.866242 11159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:44:16.876092 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 11159
I20260810 03:44:16.876272 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:44:16.876626 22205 external_mini_cluster.cc:1428] Setting key 97e322667b0d911ac733d4417f740966
Aug 10 03:44:16 dist-test-slave-ss4n krb5kdc[10731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333456, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.67@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:16.882856 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:16.868387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:34789 (local address 127.21.175.126:37933)
0810 03:44:16.868559 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:16.868564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:16.869564 (+ 1000us) server_negotiation.cc:407] Connection header received
0810 03:44:16.871143 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:16.871147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:16.871211 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:16.871300 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:16.872596 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.873197 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.874574 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.874832 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.874969 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:16.878352 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:16.878358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:16.878364 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:16.878405 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:16.880428 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:16.880958 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:16.880961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:16.880962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:16.881075 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:16.881415 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:16.881418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:16.881419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:16.881718 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:16.881843 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:16.882630 (+ 787us) server_negotiation.cc:299] Negotiation successful
0810 03:44:16.882738 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:44:16.883932 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:16.868788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:37933 (local address 127.21.175.67:34789)
0810 03:44:16.869291 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:16.869338 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:16.870906 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:16.871493 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:16.871501 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:16.871885 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:16.872414 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:16.872430 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.873345 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.873349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:16.874416 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:16.874426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.875016 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.875020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:16.875207 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:16.876069 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:16.876087 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:16.878173 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:16.880556 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:16.880561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:16.880576 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:16.880851 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:16.881166 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:16.881168 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:16.881169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:16.881316 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:16.881835 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:16.881841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:16.882120 (+ 279us) client_negotiation.cc:769] Sending connection context
0810 03:44:16.882510 (+ 390us) client_negotiation.cc:240] Negotiation successful
0810 03:44:16.882834 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":133,"threads_started":1}
I20260810 03:44:16.885217 11299 heartbeater.cc:347] Connected to a master server at 127.21.175.126:37933
I20260810 03:44:16.885568 11299 heartbeater.cc:464] Registering TS with master...
I20260810 03:44:16.886158 11299 heartbeater.cc:511] Master 127.21.175.126:37933 requested a full tablet report, sending...
I20260810 03:44:16.887352 10782 ts_manager.cc:194] Registered new tserver with Master: dcaa1a2425d1417eaba15e330b6dcf1f (127.21.175.67:39089)
I20260810 03:44:16.887912 10782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.175.67@KRBTEST.COM'} at 127.21.175.67:34789
I20260810 03:44:16.892618 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:44:16.905221 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:16.895232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.21.175.126:37933)
0810 03:44:16.895403 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:16.895407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:16.895652 (+ 245us) server_negotiation.cc:407] Connection header received
0810 03:44:16.895740 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:16.895742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:16.895787 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:16.895855 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:16.896751 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.897385 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.898455 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.898713 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.898811 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:16.900383 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:16.900387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:16.900389 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:16.900422 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:16.902905 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:16.903468 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:16.903473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:16.903475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:16.903615 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:16.904054 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:16.904058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:16.904060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:16.904349 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:16.904550 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:16.904846 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 03:44:16.904997 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:44:16.908597 10782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:44:16.912210 10782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:44:16.928148 11313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:16.921721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60752 (local address 127.21.175.67:39089)
0810 03:44:16.922393 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:16.922406 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:16.922434 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:44:16.922520 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:16.922526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:16.922754 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:16.922928 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:16.923423 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.924565 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.926510 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.927356 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.927472 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:16.927620 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:16.927859 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 03:44:16.927964 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":147,"threads_started":1}
I20260810 03:44:16.928141 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:16.921602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:39089 (local address 127.0.0.1:60752)
0810 03:44:16.922000 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:16.922051 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:16.922218 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:16.923141 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:16.923146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:16.923170 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:16.923302 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:16.923308 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.924721 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.924728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:16.926370 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:16.926383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.927495 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.927499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:16.927631 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:16.927682 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 03:44:16.927815 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 03:44:16.927960 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":10,"thread_start_us":111,"threads_started":1}
I20260810 03:44:16.928653 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:16.921800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:37331 (local address 127.0.0.1:60720)
0810 03:44:16.922952 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:16.922966 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:16.923050 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:16.923697 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:16.923700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:16.923717 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:16.923825 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:16.923832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.925253 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.925257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:16.926820 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:16.926831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.928171 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.928178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:16.928296 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:16.928330 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 03:44:16.928418 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 03:44:16.928502 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"mutex_wait_us":685,"thread_start_us":108,"threads_started":1}
I20260810 03:44:16.928951 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:16.922042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.21.175.66:37331)
0810 03:44:16.922807 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:16.922823 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:16.923146 (+ 323us) server_negotiation.cc:407] Connection header received
0810 03:44:16.923223 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:16.923229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:16.923432 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:16.923583 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:16.923960 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.925130 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.926955 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.927840 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.927944 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:16.928096 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:16.928450 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 03:44:16.928542 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":144,"threads_started":1}
I20260810 03:44:16.930004 11318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:16.922449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:37327 (local address 127.0.0.1:54342)
0810 03:44:16.923462 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:16.923481 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:16.923574 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:16.923969 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:16.923972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:16.923989 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:16.924068 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:16.924073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.925372 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.925377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:16.927043 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:16.927057 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.928616 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.928619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:16.928702 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:16.929696 (+ 994us) client_negotiation.cc:769] Sending connection context
0810 03:44:16.929792 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 03:44:16.929869 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":146,"thread_start_us":153,"threads_started":1}
I20260810 03:44:16.930141 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:16.922957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54342 (local address 127.21.175.65:37327)
0810 03:44:16.923574 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:16.923580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:16.923602 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:44:16.923677 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:16.923682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:16.923842 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:16.923991 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:16.924168 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.925253 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.927184 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:16.927992 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:16.928093 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:16.929289 (+ 1196us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:16.929863 (+ 574us) server_negotiation.cc:299] Negotiation successful
0810 03:44:16.929977 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":128,"threads_started":1}
I20260810 03:44:16.931778 11227 tablet_service.cc:1467] Processing CreateTablet for tablet df0b18ffb8f24370b899d9b4aee24f57 (DEFAULT_TABLE table=test-table [id=f31e3c5d1ae440dab0111c0aa924926e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:16.931900 11077 tablet_service.cc:1467] Processing CreateTablet for tablet df0b18ffb8f24370b899d9b4aee24f57 (DEFAULT_TABLE table=test-table [id=f31e3c5d1ae440dab0111c0aa924926e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:16.933054 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0b18ffb8f24370b899d9b4aee24f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:16.933055 11077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0b18ffb8f24370b899d9b4aee24f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:16.932958 10925 tablet_service.cc:1467] Processing CreateTablet for tablet df0b18ffb8f24370b899d9b4aee24f57 (DEFAULT_TABLE table=test-table [id=f31e3c5d1ae440dab0111c0aa924926e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:16.934047 10925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0b18ffb8f24370b899d9b4aee24f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:16.940016 11320 tablet_bootstrap.cc:492] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12: Bootstrap starting.
I20260810 03:44:16.941959 11321 tablet_bootstrap.cc:492] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f: Bootstrap starting.
I20260810 03:44:16.942014 11322 tablet_bootstrap.cc:492] T df0b18ffb8f24370b899d9b4aee24f57 P bb09884d77944656afb9043d4f5c95dd: Bootstrap starting.
I20260810 03:44:16.942780 11320 tablet_bootstrap.cc:654] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:16.944355 11320 log.cc:824] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:16.944490 11322 tablet_bootstrap.cc:654] T df0b18ffb8f24370b899d9b4aee24f57 P bb09884d77944656afb9043d4f5c95dd: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:16.944490 11321 tablet_bootstrap.cc:654] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:16.945531 11322 log.cc:824] T df0b18ffb8f24370b899d9b4aee24f57 P bb09884d77944656afb9043d4f5c95dd: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:16.945863 11321 log.cc:824] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:16.946924 11320 tablet_bootstrap.cc:492] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12: No bootstrap required, opened a new log
I20260810 03:44:16.947161 11320 ts_tablet_manager.cc:1397] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 03:44:16.947793 11322 tablet_bootstrap.cc:492] T df0b18ffb8f24370b899d9b4aee24f57 P bb09884d77944656afb9043d4f5c95dd: No bootstrap required, opened a new log
I20260810 03:44:16.948022 11322 ts_tablet_manager.cc:1397] T df0b18ffb8f24370b899d9b4aee24f57 P bb09884d77944656afb9043d4f5c95dd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:44:16.948113 11321 tablet_bootstrap.cc:492] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f: No bootstrap required, opened a new log
I20260810 03:44:16.948315 11321 ts_tablet_manager.cc:1397] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 03:44:16.951853 11320 raft_consensus.cc:348] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcaa1a2425d1417eaba15e330b6dcf1f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39089 } } peers { permanent_uuid: "bb09884d77944656afb9043d4f5c95dd" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37327 } } peers { permanent_uuid: "76d3681121774e048f635cf834fc7d12" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 37331 } }
I20260810 03:44:16.952199 11320 raft_consensus.cc:374] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:16.952268 11320 raft_consensus.cc:729] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d3681121774e048f635cf834fc7d12, State: Initialized, Role: FOLLOWER
I20260810 03:44:16.952315 11322 raft_consensus.cc:348] T df0b18ffb8f24370b899d9b4aee24f57 P bb09884d77944656afb9043d4f5c95dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcaa1a2425d1417eaba15e330b6dcf1f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39089 } } peers { permanent_uuid: "bb09884d77944656afb9043d4f5c95dd" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37327 } } peers { permanent_uuid: "76d3681121774e048f635cf834fc7d12" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 37331 } }
I20260810 03:44:16.952673 11322 raft_consensus.cc:374] T df0b18ffb8f24370b899d9b4aee24f57 P bb09884d77944656afb9043d4f5c95dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:16.952745 11322 raft_consensus.cc:729] T df0b18ffb8f24370b899d9b4aee24f57 P bb09884d77944656afb9043d4f5c95dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb09884d77944656afb9043d4f5c95dd, State: Initialized, Role: FOLLOWER
I20260810 03:44:16.952852 11320 consensus_queue.cc:260] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcaa1a2425d1417eaba15e330b6dcf1f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39089 } } peers { permanent_uuid: "bb09884d77944656afb9043d4f5c95dd" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37327 } } peers { permanent_uuid: "76d3681121774e048f635cf834fc7d12" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 37331 } }
I20260810 03:44:16.952803 11321 raft_consensus.cc:348] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcaa1a2425d1417eaba15e330b6dcf1f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39089 } } peers { permanent_uuid: "bb09884d77944656afb9043d4f5c95dd" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37327 } } peers { permanent_uuid: "76d3681121774e048f635cf834fc7d12" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 37331 } }
I20260810 03:44:16.953058 11321 raft_consensus.cc:374] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:16.953135 11321 raft_consensus.cc:729] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcaa1a2425d1417eaba15e330b6dcf1f, State: Initialized, Role: FOLLOWER
I20260810 03:44:16.953243 11322 consensus_queue.cc:260] T df0b18ffb8f24370b899d9b4aee24f57 P bb09884d77944656afb9043d4f5c95dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcaa1a2425d1417eaba15e330b6dcf1f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39089 } } peers { permanent_uuid: "bb09884d77944656afb9043d4f5c95dd" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37327 } } peers { permanent_uuid: "76d3681121774e048f635cf834fc7d12" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 37331 } }
I20260810 03:44:16.953621 11321 consensus_queue.cc:260] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcaa1a2425d1417eaba15e330b6dcf1f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39089 } } peers { permanent_uuid: "bb09884d77944656afb9043d4f5c95dd" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37327 } } peers { permanent_uuid: "76d3681121774e048f635cf834fc7d12" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 37331 } }
I20260810 03:44:16.954118 11000 heartbeater.cc:503] Master 127.21.175.126:37933 was elected leader, sending a full tablet report...
I20260810 03:44:16.954363 11322 ts_tablet_manager.cc:1428] T df0b18ffb8f24370b899d9b4aee24f57 P bb09884d77944656afb9043d4f5c95dd: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260810 03:44:16.954613 11321 ts_tablet_manager.cc:1428] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 03:44:16.954945 11299 heartbeater.cc:503] Master 127.21.175.126:37933 was elected leader, sending a full tablet report...
I20260810 03:44:16.955344 11320 ts_tablet_manager.cc:1428] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 03:44:16.955756 11152 heartbeater.cc:503] Master 127.21.175.126:37933 was elected leader, sending a full tablet report...
I20260810 03:44:17.052987 11328 raft_consensus.cc:482] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:44:17.053277 11328 raft_consensus.cc:504] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcaa1a2425d1417eaba15e330b6dcf1f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39089 } } peers { permanent_uuid: "bb09884d77944656afb9043d4f5c95dd" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37327 } } peers { permanent_uuid: "76d3681121774e048f635cf834fc7d12" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 37331 } }
I20260810 03:44:17.054610 11328 leader_election.cc:288] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb09884d77944656afb9043d4f5c95dd (127.21.175.65:37327), 76d3681121774e048f635cf834fc7d12 (127.21.175.66:37331)
I20260810 03:44:17.060272 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:17.055049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:36081 (local address 127.21.175.66:37331)
0810 03:44:17.055196 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:17.055200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:17.055579 (+ 379us) server_negotiation.cc:407] Connection header received
0810 03:44:17.055736 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:17.055739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:17.055784 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:17.055849 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:17.056167 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.056873 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.058771 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.059414 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.059532 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:17.059588 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:17.060058 (+ 470us) server_negotiation.cc:299] Negotiation successful
0810 03:44:17.060137 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:44:17.060272 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:17.055152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:58293 (local address 127.21.175.65:37327)
0810 03:44:17.055302 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:17.055308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:17.055349 (+ 41us) server_negotiation.cc:407] Connection header received
0810 03:44:17.055419 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:17.055422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:17.055469 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:17.055563 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:17.055935 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.056725 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.058771 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.059398 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.059532 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:17.059599 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:17.060058 (+ 459us) server_negotiation.cc:299] Negotiation successful
0810 03:44:17.060137 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:44:17.060316 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:17.054919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:37327 (local address 127.21.175.67:58293)
0810 03:44:17.055098 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:17.055120 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:17.055237 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:17.055614 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:17.055617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:17.055636 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:17.055781 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:17.055786 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.056859 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.056862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:17.058613 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:17.058626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.059577 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.059582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:17.059811 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:17.059935 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 03:44:17.060025 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 03:44:17.060119 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":275}
I20260810 03:44:17.060325 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:17.054919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:37331 (local address 127.21.175.67:36081)
0810 03:44:17.055459 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:17.055471 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:17.055608 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:17.055890 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:17.055895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:17.055922 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:17.056040 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:17.056045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.056969 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.056973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:17.058613 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:17.058626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.059577 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.059582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:17.059806 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:17.059932 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 03:44:17.060027 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 03:44:17.060119 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":320,"thread_start_us":150,"threads_started":1}
I20260810 03:44:17.061125 11101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b18ffb8f24370b899d9b4aee24f57" candidate_uuid: "dcaa1a2425d1417eaba15e330b6dcf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d3681121774e048f635cf834fc7d12" is_pre_election: true
I20260810 03:44:17.061121 10946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b18ffb8f24370b899d9b4aee24f57" candidate_uuid: "dcaa1a2425d1417eaba15e330b6dcf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb09884d77944656afb9043d4f5c95dd" is_pre_election: true
I20260810 03:44:17.061522 11101 raft_consensus.cc:2454] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcaa1a2425d1417eaba15e330b6dcf1f in term 0.
I20260810 03:44:17.061553 10946 raft_consensus.cc:2454] T df0b18ffb8f24370b899d9b4aee24f57 P bb09884d77944656afb9043d4f5c95dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcaa1a2425d1417eaba15e330b6dcf1f in term 0.
I20260810 03:44:17.062125 11179 leader_election.cc:302] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 76d3681121774e048f635cf834fc7d12, bb09884d77944656afb9043d4f5c95dd, dcaa1a2425d1417eaba15e330b6dcf1f; no voters:
I20260810 03:44:17.062464 11328 raft_consensus.cc:2781] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:44:17.062548 11328 raft_consensus.cc:482] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:44:17.062624 11328 raft_consensus.cc:3037] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:17.063720 11328 raft_consensus.cc:504] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcaa1a2425d1417eaba15e330b6dcf1f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39089 } } peers { permanent_uuid: "bb09884d77944656afb9043d4f5c95dd" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37327 } } peers { permanent_uuid: "76d3681121774e048f635cf834fc7d12" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 37331 } }
I20260810 03:44:17.064134 11328 leader_election.cc:288] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [CANDIDATE]: Term 1 election: Requested vote from peers bb09884d77944656afb9043d4f5c95dd (127.21.175.65:37327), 76d3681121774e048f635cf834fc7d12 (127.21.175.66:37331)
I20260810 03:44:17.064595 11101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b18ffb8f24370b899d9b4aee24f57" candidate_uuid: "dcaa1a2425d1417eaba15e330b6dcf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d3681121774e048f635cf834fc7d12"
I20260810 03:44:17.064595 10946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b18ffb8f24370b899d9b4aee24f57" candidate_uuid: "dcaa1a2425d1417eaba15e330b6dcf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb09884d77944656afb9043d4f5c95dd"
I20260810 03:44:17.064754 11101 raft_consensus.cc:3037] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:17.064775 10946 raft_consensus.cc:3037] T df0b18ffb8f24370b899d9b4aee24f57 P bb09884d77944656afb9043d4f5c95dd [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:17.066311 11101 raft_consensus.cc:2454] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcaa1a2425d1417eaba15e330b6dcf1f in term 1.
I20260810 03:44:17.066311 10946 raft_consensus.cc:2454] T df0b18ffb8f24370b899d9b4aee24f57 P bb09884d77944656afb9043d4f5c95dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcaa1a2425d1417eaba15e330b6dcf1f in term 1.
I20260810 03:44:17.066697 11179 leader_election.cc:302] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb09884d77944656afb9043d4f5c95dd, dcaa1a2425d1417eaba15e330b6dcf1f; no voters:
I20260810 03:44:17.066912 11328 raft_consensus.cc:2781] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:17.067226 11328 raft_consensus.cc:686] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [term 1 LEADER]: Becoming Leader. State: Replica: dcaa1a2425d1417eaba15e330b6dcf1f, State: Running, Role: LEADER
I20260810 03:44:17.067685 11328 consensus_queue.cc:237] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcaa1a2425d1417eaba15e330b6dcf1f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39089 } } peers { permanent_uuid: "bb09884d77944656afb9043d4f5c95dd" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37327 } } peers { permanent_uuid: "76d3681121774e048f635cf834fc7d12" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 37331 } }
I20260810 03:44:17.071501 10781 catalog_manager.cc:5495] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f reported cstate change: term changed from 0 to 1, leader changed from <none> to dcaa1a2425d1417eaba15e330b6dcf1f (127.21.175.67). New cstate: current_term: 1 leader_uuid: "dcaa1a2425d1417eaba15e330b6dcf1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcaa1a2425d1417eaba15e330b6dcf1f" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 39089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb09884d77944656afb9043d4f5c95dd" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76d3681121774e048f635cf834fc7d12" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 37331 } health_report { overall_health: UNKNOWN } } }
W20260810 03:44:17.117579 11300 tablet.cc:2369] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:44:17.120438 11313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:17.116507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.21.175.67:39089)
0810 03:44:17.116664 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:17.116670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:17.116687 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:44:17.116789 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:17.116792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:17.116850 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:17.116971 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:44:17.117264 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.118054 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.119041 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.119248 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.119326 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:17.119648 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:44:17.120057 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:44:17.120152 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:17.120263 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 03:44:17.120314 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 03:44:17.122455 11153 tablet.cc:2369] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:44:17.126984 11101 raft_consensus.cc:1261] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12 [term 1 FOLLOWER]: Refusing update from remote peer dcaa1a2425d1417eaba15e330b6dcf1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:44:17.127050 10946 raft_consensus.cc:1261] T df0b18ffb8f24370b899d9b4aee24f57 P bb09884d77944656afb9043d4f5c95dd [term 1 FOLLOWER]: Refusing update from remote peer dcaa1a2425d1417eaba15e330b6dcf1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:44:17.127836 11328 consensus_queue.cc:1035] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb09884d77944656afb9043d4f5c95dd" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 37327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:44:17.128064 11331 consensus_queue.cc:1035] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d3681121774e048f635cf834fc7d12" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 37331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 03:44:17.128842 11001 tablet.cc:2369] T df0b18ffb8f24370b899d9b4aee24f57 P bb09884d77944656afb9043d4f5c95dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:44:17.136682 11336 mvcc.cc:205] Tried to move back new op lower bound from 7316821840383852544 to 7316821840152354816. Current Snapshot: MvccSnapshot[applied={T|T < 7316821840383852544}]
I20260810 03:44:17.147046 10781 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:44:17.147239 10781 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:44:17.149976 10781 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c466747caaa442ea49d569f7a9e5396: Sending DeleteTablet for 3 replicas of tablet df0b18ffb8f24370b899d9b4aee24f57
I20260810 03:44:17.150997 11077 tablet_service.cc:1514] Processing DeleteTablet for tablet df0b18ffb8f24370b899d9b4aee24f57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:17 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:60720
I20260810 03:44:17.151140 10925 tablet_service.cc:1514] Processing DeleteTablet for tablet df0b18ffb8f24370b899d9b4aee24f57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:17 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:54342
I20260810 03:44:17.151546 11345 tablet_replica.cc:330] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12: stopping tablet replica
I20260810 03:44:17.151566 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet df0b18ffb8f24370b899d9b4aee24f57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:17 UTC) from {username='kudu', principal='kudu/127.21.175.126@KRBTEST.COM'} at 127.0.0.1:60752
I20260810 03:44:17.151907 11345 raft_consensus.cc:2229] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:44:17.152169 11347 tablet_replica.cc:330] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f: stopping tablet replica
I20260810 03:44:17.152326 11345 raft_consensus.cc:2258] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:44:17.152751 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 10855
I20260810 03:44:17.156037 11345 ts_tablet_manager.cc:1905] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:44:17.159556 11345 ts_tablet_manager.cc:1918] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:44:17.159670 11345 log.cc:1196] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/df0b18ffb8f24370b899d9b4aee24f57
I20260810 03:44:17.160048 11345 ts_tablet_manager.cc:1939] T df0b18ffb8f24370b899d9b4aee24f57 P 76d3681121774e048f635cf834fc7d12: Deleting consensus metadata
I20260810 03:44:17.161666 10767 catalog_manager.cc:4841] TS 76d3681121774e048f635cf834fc7d12 (127.21.175.66:37331): tablet df0b18ffb8f24370b899d9b4aee24f57 (table test-table [id=f31e3c5d1ae440dab0111c0aa924926e]) successfully deleted
I20260810 03:44:17.152613 11347 raft_consensus.cc:2229] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:44:17.163125 11347 raft_consensus.cc:2258] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:44:17.167948 11347 ts_tablet_manager.cc:1905] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:44:17.170907 11347 ts_tablet_manager.cc:1918] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:44:17.171020 11347 log.cc:1196] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/df0b18ffb8f24370b899d9b4aee24f57
I20260810 03:44:17.171360 11347 ts_tablet_manager.cc:1939] T df0b18ffb8f24370b899d9b4aee24f57 P dcaa1a2425d1417eaba15e330b6dcf1f: Deleting consensus metadata
W20260810 03:44:17.172156 10766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.175.65:37327 (error 108)
I20260810 03:44:17.172775 10766 catalog_manager.cc:4841] TS dcaa1a2425d1417eaba15e330b6dcf1f (127.21.175.67:39089): tablet df0b18ffb8f24370b899d9b4aee24f57 (table test-table [id=f31e3c5d1ae440dab0111c0aa924926e]) successfully deleted
I20260810 03:44:17.173434 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 11005
I20260810 03:44:17.173503 11318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:17.173129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:37327 (local address 127.0.0.1:54352)
0810 03:44:17.173305 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:17.173389 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 03:44:17.174867 10766 catalog_manager.cc:4584] TS bb09884d77944656afb9043d4f5c95dd (127.21.175.65:37327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet df0b18ffb8f24370b899d9b4aee24f57: Network error: Client connection negotiation failed: client connection to 127.21.175.65:37327: connect: Connection refused (error 111)
I20260810 03:44:17.180913 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 11159
I20260810 03:44:17.189742 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 10747
2026-08-10T03:44:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2544 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T03:44:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:17Z Disabled control of system clock
I20260810 03:44:17.224714 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:46597
--webserver_interface=127.21.175.126
--webserver_port=0
--builtin_ntp_servers=127.21.175.84:36271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:46597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:44:17.342444 11353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:17.342756 11353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:17.342792 11353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:17.346917 11353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:17.346990 11353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:17.347007 11353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:17.347024 11353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:17.347039 11353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:17.352192 11353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:36271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:46597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:46597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:17.353340 11353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:17.354820 11353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:17.364225 11359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:17.364264 11360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:17.364352 11353 server_base.cc:1034] running on GCE node
W20260810 03:44:17.364225 11363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:17.365160 11353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:17.366416 11353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:17.367671 11353 hybrid_clock.cc:648] HybridClock initialized: now 1786333457367641 us; error 43 us; skew 500 ppm
I20260810 03:44:17.369979 11353 webserver.cc:460] Webserver started at http://127.21.175.126:35047/ using document root <none> and password file <none>
I20260810 03:44:17.370677 11353 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:17.370808 11353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:17.371079 11353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:17.373409 11353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "0eacbc4522fe4a46b132523a58eef3fe"
format_stamp: "Formatted at 2026-08-10 03:44:17 on dist-test-slave-ss4n"
server_key: "8543c843a83466151d90fa93ff27a38b"
server_key_iv: "31cfe8875885779dd8c6f5d713d8b90d"
server_key_version: "encryptionkey@0"
I20260810 03:44:17.374131 11353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "0eacbc4522fe4a46b132523a58eef3fe"
format_stamp: "Formatted at 2026-08-10 03:44:17 on dist-test-slave-ss4n"
server_key: "8543c843a83466151d90fa93ff27a38b"
server_key_iv: "31cfe8875885779dd8c6f5d713d8b90d"
server_key_version: "encryptionkey@0"
I20260810 03:44:17.378422 11353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:44:17.381330 11370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:17.383009 11353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:44:17.383172 11353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "0eacbc4522fe4a46b132523a58eef3fe"
format_stamp: "Formatted at 2026-08-10 03:44:17 on dist-test-slave-ss4n"
server_key: "8543c843a83466151d90fa93ff27a38b"
server_key_iv: "31cfe8875885779dd8c6f5d713d8b90d"
server_key_version: "encryptionkey@0"
I20260810 03:44:17.383303 11353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:17.395339 11353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:17.396116 11353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:17.396338 11353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:17.404136 11353 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:46597
I20260810 03:44:17.404163 11429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:46597 every 8 connection(s)
I20260810 03:44:17.405179 11353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:17.408672 11430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:17.411550 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 11353
I20260810 03:44:17.411708 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:17.412005 22205 external_mini_cluster.cc:1428] Setting key af69e269821e4c3f37bad0b9d50d89a1
I20260810 03:44:17.415336 11430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe: Bootstrap starting.
I20260810 03:44:17.418247 11430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:17.419188 11430 log.cc:824] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:17.421777 11430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe: No bootstrap required, opened a new log
I20260810 03:44:17.422851 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:17.413557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52284 (local address 127.21.175.126:46597)
0810 03:44:17.414247 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:17.414263 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:17.414301 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:44:17.414981 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:17.414999 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:17.415427 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:17.415793 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:17.416186 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.417307 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.418697 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.419229 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.419438 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:17.420466 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:17.420473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:17.420492 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:17.421033 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:17.421133 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:17.421162 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:17.421274 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:17.421449 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 03:44:17.421803 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":138,"threads_started":1}
I20260810 03:44:17.426416 11430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eacbc4522fe4a46b132523a58eef3fe" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46597 } }
I20260810 03:44:17.426672 11430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:17.426712 11430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eacbc4522fe4a46b132523a58eef3fe, State: Initialized, Role: FOLLOWER
I20260810 03:44:17.427101 11430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eacbc4522fe4a46b132523a58eef3fe" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46597 } }
I20260810 03:44:17.427224 11430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:17.427268 11430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:17.427345 11430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:17.428537 11430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eacbc4522fe4a46b132523a58eef3fe" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46597 } }
I20260810 03:44:17.428874 11430 leader_election.cc:302] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eacbc4522fe4a46b132523a58eef3fe; no voters:
I20260810 03:44:17.429175 11430 leader_election.cc:288] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:17.429342 11435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:17.429670 11435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [term 1 LEADER]: Becoming Leader. State: Replica: 0eacbc4522fe4a46b132523a58eef3fe, State: Running, Role: LEADER
I20260810 03:44:17.430091 11435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eacbc4522fe4a46b132523a58eef3fe" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46597 } }
I20260810 03:44:17.430640 11430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:17.431998 11436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eacbc4522fe4a46b132523a58eef3fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eacbc4522fe4a46b132523a58eef3fe" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46597 } } }
I20260810 03:44:17.432034 11437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eacbc4522fe4a46b132523a58eef3fe. Latest consensus state: current_term: 1 leader_uuid: "0eacbc4522fe4a46b132523a58eef3fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eacbc4522fe4a46b132523a58eef3fe" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 46597 } } }
I20260810 03:44:17.432173 11436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:17.432179 11437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:17.432580 11445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:17.435895 11445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:17.442260 11445 catalog_manager.cc:1347] Generated new cluster ID: 6527248fe9a84cb6b3b8fd1a97513dcb
I20260810 03:44:17.442378 11445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:17.451058 11445 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:17.452042 11445 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:17.462726 11445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe: Generated new TSK 0
I20260810 03:44:17.463491 11445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:17.467999 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.65:0
--local_ip_for_outbound_sockets=127.21.175.65
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46597
--builtin_ntp_servers=127.21.175.84:36271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:44:17.614768 11456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:17.615049 11456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:17.615118 11456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:17.619378 11456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:17.619444 11456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:17.619503 11456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.65
I20260810 03:44:17.624485 11456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:36271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.175.65
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.65
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:17.625771 11456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:17.627252 11456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:17.637640 11462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:17.637651 11461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:17.637748 11464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:17.638092 11456 server_base.cc:1034] running on GCE node
I20260810 03:44:17.638540 11456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:17.639226 11456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:17.640523 11456 hybrid_clock.cc:648] HybridClock initialized: now 1786333457640496 us; error 41 us; skew 500 ppm
I20260810 03:44:17.642889 11456 webserver.cc:460] Webserver started at http://127.21.175.65:34325/ using document root <none> and password file <none>
I20260810 03:44:17.643519 11456 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:17.643669 11456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:17.643932 11456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:17.646188 11456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-0/data/instance:
uuid: "b5cbc627f7cd43438dd5035874204ded"
format_stamp: "Formatted at 2026-08-10 03:44:17 on dist-test-slave-ss4n"
server_key: "4142edcb573bca24fbafaa8633bd6925"
server_key_iv: "7139b7f64980d4624bc8bca7b54d2289"
server_key_version: "encryptionkey@0"
I20260810 03:44:17.646808 11456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance:
uuid: "b5cbc627f7cd43438dd5035874204ded"
format_stamp: "Formatted at 2026-08-10 03:44:17 on dist-test-slave-ss4n"
server_key: "4142edcb573bca24fbafaa8633bd6925"
server_key_iv: "7139b7f64980d4624bc8bca7b54d2289"
server_key_version: "encryptionkey@0"
I20260810 03:44:17.650775 11456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:44:17.653357 11472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:17.654721 11456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:44:17.654860 11456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-0/wal
uuid: "b5cbc627f7cd43438dd5035874204ded"
format_stamp: "Formatted at 2026-08-10 03:44:17 on dist-test-slave-ss4n"
server_key: "4142edcb573bca24fbafaa8633bd6925"
server_key_iv: "7139b7f64980d4624bc8bca7b54d2289"
server_key_version: "encryptionkey@0"
I20260810 03:44:17.654983 11456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:17.686455 11456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:17.687346 11456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:17.687602 11456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:17.688293 11456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:17.689375 11456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:17.689455 11456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:17.689558 11456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:17.689611 11456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:17.699570 11456 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.65:40963
I20260810 03:44:17.699585 11601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.65:40963 every 8 connection(s)
I20260810 03:44:17.700639 11456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-0/data/info.pb
I20260810 03:44:17.708011 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 11456
I20260810 03:44:17.708137 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-0/wal/instance
I20260810 03:44:17.708436 22205 external_mini_cluster.cc:1428] Setting key 6b68c7e17d11e00ed18580ac1997430f
I20260810 03:44:17.710961 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.66:0
--local_ip_for_outbound_sockets=127.21.175.66
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46597
--builtin_ntp_servers=127.21.175.84:36271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:44:17.712033 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:17.702787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.65:55783 (local address 127.21.175.126:46597)
0810 03:44:17.702997 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:17.703003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:17.704243 (+ 1240us) server_negotiation.cc:407] Connection header received
0810 03:44:17.705875 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:17.705878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:17.705932 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:17.706021 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:17.707150 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.707819 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.709363 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.709755 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.709889 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:17.711453 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:17.711462 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:17.711464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:17.711479 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:17.711501 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:17.711511 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:17.711602 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:17.711791 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 03:44:17.711864 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 03:44:17.713008 11604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:17.703264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46597 (local address 127.21.175.65:55783)
0810 03:44:17.704062 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:17.704112 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:17.705601 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:17.706267 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:17.706274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:17.706376 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:17.706990 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:17.707004 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.707983 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.707986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:17.709211 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:17.709225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.710032 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.710037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:17.710253 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:17.711047 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:44:17.711049 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:17.711131 (+ 82us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:44:17.711242 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:17.711606 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:17.711620 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:17.711629 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 03:44:17.711791 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 03:44:17.712141 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":119,"threads_started":1}
I20260810 03:44:17.714646 11602 heartbeater.cc:347] Connected to a master server at 127.21.175.126:46597
I20260810 03:44:17.714959 11602 heartbeater.cc:464] Registering TS with master...
I20260810 03:44:17.715595 11602 heartbeater.cc:511] Master 127.21.175.126:46597 requested a full tablet report, sending...
I20260810 03:44:17.717656 11391 ts_manager.cc:194] Registered new tserver with Master: b5cbc627f7cd43438dd5035874204ded (127.21.175.65:40963)
I20260810 03:44:17.719684 11391 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.175.65:55783
W20260810 03:44:17.834112 11605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:17.834435 11605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:17.834477 11605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:17.838671 11605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:17.838732 11605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:17.838788 11605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.66
I20260810 03:44:17.843809 11605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:36271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.175.66
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.66
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:17.844944 11605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:17.846423 11605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:17.856555 11613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:17.856508 11610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:17.856575 11611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:17.856714 11605 server_base.cc:1034] running on GCE node
I20260810 03:44:17.857332 11605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:17.858135 11605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:17.859365 11605 hybrid_clock.cc:648] HybridClock initialized: now 1786333457859338 us; error 35 us; skew 500 ppm
I20260810 03:44:17.861433 11605 webserver.cc:460] Webserver started at http://127.21.175.66:40963/ using document root <none> and password file <none>
I20260810 03:44:17.862131 11605 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:17.862213 11605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:17.862389 11605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:17.864537 11605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/data/instance:
uuid: "a83439d0b514471091183cc3f85913ef"
format_stamp: "Formatted at 2026-08-10 03:44:17 on dist-test-slave-ss4n"
server_key: "7c3783d8299cc5d25a57d1bd6d70d206"
server_key_iv: "2dadfd769f61cdfbf7f8cdfcc8c5d339"
server_key_version: "encryptionkey@0"
I20260810 03:44:17.865082 11605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance:
uuid: "a83439d0b514471091183cc3f85913ef"
format_stamp: "Formatted at 2026-08-10 03:44:17 on dist-test-slave-ss4n"
server_key: "7c3783d8299cc5d25a57d1bd6d70d206"
server_key_iv: "2dadfd769f61cdfbf7f8cdfcc8c5d339"
server_key_version: "encryptionkey@0"
I20260810 03:44:17.868893 11605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:44:17.871491 11620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:17.872891 11605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 03:44:17.873027 11605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/wal
uuid: "a83439d0b514471091183cc3f85913ef"
format_stamp: "Formatted at 2026-08-10 03:44:17 on dist-test-slave-ss4n"
server_key: "7c3783d8299cc5d25a57d1bd6d70d206"
server_key_iv: "2dadfd769f61cdfbf7f8cdfcc8c5d339"
server_key_version: "encryptionkey@0"
I20260810 03:44:17.873152 11605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:17.890257 11605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:17.891009 11605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:17.891235 11605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:17.891875 11605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:17.892922 11605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:17.892972 11605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:17.893014 11605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:17.893069 11605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:17.903769 11605 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.66:43913
I20260810 03:44:17.903882 11750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.66:43913 every 8 connection(s)
I20260810 03:44:17.904824 11605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/data/info.pb
I20260810 03:44:17.909961 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 11605
I20260810 03:44:17.910095 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/wal/instance
I20260810 03:44:17.910414 22205 external_mini_cluster.cc:1428] Setting key 561da9f203b6eff8707dfb97475af82c
I20260810 03:44:17.912789 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.175.67:0
--local_ip_for_outbound_sockets=127.21.175.67
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46597
--builtin_ntp_servers=127.21.175.84:36271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:44:17.916796 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:17.906840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.66:45811 (local address 127.21.175.126:46597)
0810 03:44:17.907051 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:17.907056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:17.908150 (+ 1094us) server_negotiation.cc:407] Connection header received
0810 03:44:17.909708 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:17.909711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:17.909766 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:17.909851 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:17.911147 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.911873 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.913779 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.914295 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.914446 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:17.916097 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:17.916104 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:17.916111 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:17.916130 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:17.916160 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:17.916174 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:17.916270 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:17.916504 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 03:44:17.916580 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260810 03:44:17.917950 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:17.907311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46597 (local address 127.21.175.66:45811)
0810 03:44:17.907990 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:17.908031 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:17.909344 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:17.910103 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:17.910116 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:17.910234 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:17.910942 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:17.910959 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.912025 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.912028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:17.913606 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:17.913622 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:17.914531 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:17.914539 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:17.914871 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:17.915684 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:44:17.915687 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:17.915769 (+ 82us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:44:17.915879 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:17.916272 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:17.916309 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:17.916318 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 03:44:17.916474 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 03:44:17.916864 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
I20260810 03:44:17.919962 11752 heartbeater.cc:347] Connected to a master server at 127.21.175.126:46597
I20260810 03:44:17.920223 11752 heartbeater.cc:464] Registering TS with master...
I20260810 03:44:17.920876 11752 heartbeater.cc:511] Master 127.21.175.126:46597 requested a full tablet report, sending...
I20260810 03:44:17.922367 11391 ts_manager.cc:194] Registered new tserver with Master: a83439d0b514471091183cc3f85913ef (127.21.175.66:43913)
I20260810 03:44:17.923308 11391 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.175.66:45811
W20260810 03:44:18.038571 11755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:18.038890 11755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:18.038931 11755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:18.043248 11755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:18.043326 11755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:18.043416 11755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.175.67
I20260810 03:44:18.048592 11755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:36271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.175.67
--webserver_port=0
--tserver_master_addrs=127.21.175.126:46597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.175.67
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:18.049803 11755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:18.051172 11755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:18.061209 11761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:18.061210 11762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:18.061210 11764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:18.061779 11755 server_base.cc:1034] running on GCE node
I20260810 03:44:18.062173 11755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:18.062799 11755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:18.063969 11755 hybrid_clock.cc:648] HybridClock initialized: now 1786333458063930 us; error 52 us; skew 500 ppm
I20260810 03:44:18.066001 11755 webserver.cc:460] Webserver started at http://127.21.175.67:33155/ using document root <none> and password file <none>
I20260810 03:44:18.066604 11755 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:18.066669 11755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:18.066919 11755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:18.069092 11755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/data/instance:
uuid: "ae6eeddeb6e14a3eb7a279fa456b196e"
format_stamp: "Formatted at 2026-08-10 03:44:18 on dist-test-slave-ss4n"
server_key: "c412caf7dc2843281585f9af2568acbd"
server_key_iv: "33a608205617bf4daa2e15ba0103b398"
server_key_version: "encryptionkey@0"
I20260810 03:44:18.069705 11755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance:
uuid: "ae6eeddeb6e14a3eb7a279fa456b196e"
format_stamp: "Formatted at 2026-08-10 03:44:18 on dist-test-slave-ss4n"
server_key: "c412caf7dc2843281585f9af2568acbd"
server_key_iv: "33a608205617bf4daa2e15ba0103b398"
server_key_version: "encryptionkey@0"
I20260810 03:44:18.073670 11755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:44:18.076349 11770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:18.077876 11755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:44:18.078011 11755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/wal
uuid: "ae6eeddeb6e14a3eb7a279fa456b196e"
format_stamp: "Formatted at 2026-08-10 03:44:18 on dist-test-slave-ss4n"
server_key: "c412caf7dc2843281585f9af2568acbd"
server_key_iv: "33a608205617bf4daa2e15ba0103b398"
server_key_version: "encryptionkey@0"
I20260810 03:44:18.078121 11755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:18.097890 11755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:18.098675 11755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:18.098889 11755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:18.099536 11755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:44:18.100540 11755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:44:18.100596 11755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:18.100673 11755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:44:18.100713 11755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:18.110078 11755 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.67:32797
I20260810 03:44:18.110157 11897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.67:32797 every 8 connection(s)
I20260810 03:44:18.111119 11755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/data/info.pb
I20260810 03:44:18.115067 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 11755
I20260810 03:44:18.115164 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/wal/instance
I20260810 03:44:18.115554 22205 external_mini_cluster.cc:1428] Setting key ee38e0ddf60269023fafd3850f428697
I20260810 03:44:18.121644 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:18.113183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:56811 (local address 127.21.175.126:46597)
0810 03:44:18.113336 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:18.113340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:18.114725 (+ 1385us) server_negotiation.cc:407] Connection header received
0810 03:44:18.116493 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:18.116497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:18.116568 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:18.116694 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:18.117699 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.118236 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.119424 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.119675 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.119787 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:18.121053 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:18.121059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:18.121067 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:18.121080 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:18.121103 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:18.121112 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:18.121196 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:18.121366 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 03:44:18.121432 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 03:44:18.122524 11900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:18.113754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:46597 (local address 127.21.175.67:56811)
0810 03:44:18.114558 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:18.114604 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:18.116273 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:18.116883 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:18.116891 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:18.116982 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:18.117561 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:18.117577 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.118369 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.118373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:18.119288 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:18.119298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.119845 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.119849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:18.120026 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:18.120760 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:44:18.120770 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:18.120814 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:44:18.120883 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:18.121212 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:18.121224 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:18.121235 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 03:44:18.121366 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 03:44:18.121704 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":116,"threads_started":1}
I20260810 03:44:18.123690 11898 heartbeater.cc:347] Connected to a master server at 127.21.175.126:46597
I20260810 03:44:18.123915 11898 heartbeater.cc:464] Registering TS with master...
I20260810 03:44:18.124425 11898 heartbeater.cc:511] Master 127.21.175.126:46597 requested a full tablet report, sending...
I20260810 03:44:18.125445 11391 ts_manager.cc:194] Registered new tserver with Master: ae6eeddeb6e14a3eb7a279fa456b196e (127.21.175.67:32797)
I20260810 03:44:18.126082 11391 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.175.67:56811
I20260810 03:44:18.130064 22205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:44:18.136323 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:18.132268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52298 (local address 127.21.175.126:46597)
0810 03:44:18.132469 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:18.132473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:18.132616 (+ 143us) server_negotiation.cc:407] Connection header received
0810 03:44:18.132755 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:18.132758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:18.132810 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:18.132896 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:18.133207 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.133788 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.134668 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.134909 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.135035 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:18.135954 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:18.135957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:18.135959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:18.135970 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:18.135986 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:18.135994 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:18.136052 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:18.136164 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 03:44:18.136216 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:44:18.139549 11391 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:44:18.142768 11391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:44:18.159150 11912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:18.151997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:40963 (local address 127.0.0.1:35330)
0810 03:44:18.152964 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:18.152980 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:18.153088 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:18.153931 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:18.153938 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:18.153977 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:18.154217 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:18.154226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.155645 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.155649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:18.157165 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:18.157177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.158522 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.158526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:18.158648 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:18.158725 (+ 77us) client_negotiation.cc:769] Sending connection context
0810 03:44:18.158827 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 03:44:18.158967 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":40,"spinlock_wait_cycles":5760,"thread_start_us":109,"threads_started":1}
I20260810 03:44:18.159165 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:18.152317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.21.175.65:40963)
0810 03:44:18.153052 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:18.153065 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:18.153094 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:44:18.153285 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:18.153292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:18.153570 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:18.153741 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:18.154369 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.155520 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.157329 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.158364 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.158502 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:18.158791 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:18.158859 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 03:44:18.158949 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":118,"threads_started":1}
I20260810 03:44:18.159770 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:18.151987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.67:32797 (local address 127.0.0.1:45652)
0810 03:44:18.152857 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:18.152914 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:18.153084 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:18.153942 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:18.153946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:18.153979 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:18.154476 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:18.154483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.156038 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.156041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:18.157270 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:18.157281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.159378 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.159385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:18.159490 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:18.159514 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:44:18.159595 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 03:44:18.159671 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":339,"spinlock_wait_cycles":8064,"thread_start_us":106,"threads_started":1}
I20260810 03:44:18.159950 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:18.152116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45652 (local address 127.21.175.67:32797)
0810 03:44:18.152841 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:18.152854 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:18.153269 (+ 415us) server_negotiation.cc:407] Connection header received
0810 03:44:18.153351 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:18.153355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:18.153570 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:18.153741 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:18.154618 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.155898 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.157414 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.158365 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.158495 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:18.158718 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:18.159660 (+ 942us) server_negotiation.cc:299] Negotiation successful
0810 03:44:18.159782 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":123,"threads_started":1}
I20260810 03:44:18.160485 11916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:18.152850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:43913 (local address 127.0.0.1:58642)
0810 03:44:18.154060 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:18.154075 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:18.154168 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:18.155101 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:18.155105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:18.155126 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:18.155219 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:18.155225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.156635 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.156642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:18.157828 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:18.157838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.160028 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.160034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:18.160151 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:18.160179 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 03:44:18.160253 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 03:44:18.160329 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"thread_start_us":134,"threads_started":1}
I20260810 03:44:18.160953 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:18.153349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58642 (local address 127.21.175.66:43913)
0810 03:44:18.153746 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:18.153765 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:18.154611 (+ 846us) server_negotiation.cc:407] Connection header received
0810 03:44:18.154712 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:18.154717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:18.154925 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:18.155105 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:18.155354 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.156503 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.157966 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.158620 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.158798 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:18.159023 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:18.160657 (+ 1634us) server_negotiation.cc:299] Negotiation successful
0810 03:44:18.160781 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":231,"threads_started":1}
I20260810 03:44:18.163058 11528 tablet_service.cc:1467] Processing CreateTablet for tablet 322a31f2bb60424b9acaa9faf838abc1 (DEFAULT_TABLE table=test-table [id=a3c00f56708c4b099e685ddd5637b7a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:18.163058 11823 tablet_service.cc:1467] Processing CreateTablet for tablet 322a31f2bb60424b9acaa9faf838abc1 (DEFAULT_TABLE table=test-table [id=a3c00f56708c4b099e685ddd5637b7a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:18.164317 11823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 322a31f2bb60424b9acaa9faf838abc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:18.164317 11528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 322a31f2bb60424b9acaa9faf838abc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:18.167892 11676 tablet_service.cc:1467] Processing CreateTablet for tablet 322a31f2bb60424b9acaa9faf838abc1 (DEFAULT_TABLE table=test-table [id=a3c00f56708c4b099e685ddd5637b7a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:44:18.168882 11676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 322a31f2bb60424b9acaa9faf838abc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:18.172734 11918 tablet_bootstrap.cc:492] T 322a31f2bb60424b9acaa9faf838abc1 P b5cbc627f7cd43438dd5035874204ded: Bootstrap starting.
I20260810 03:44:18.174340 11919 tablet_bootstrap.cc:492] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e: Bootstrap starting.
I20260810 03:44:18.175990 11918 tablet_bootstrap.cc:654] T 322a31f2bb60424b9acaa9faf838abc1 P b5cbc627f7cd43438dd5035874204ded: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:18.176697 11919 tablet_bootstrap.cc:654] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:18.177183 11918 log.cc:824] T 322a31f2bb60424b9acaa9faf838abc1 P b5cbc627f7cd43438dd5035874204ded: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:18.177587 11919 log.cc:824] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:18.177902 11920 tablet_bootstrap.cc:492] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef: Bootstrap starting.
I20260810 03:44:18.179600 11919 tablet_bootstrap.cc:492] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e: No bootstrap required, opened a new log
I20260810 03:44:18.179663 11918 tablet_bootstrap.cc:492] T 322a31f2bb60424b9acaa9faf838abc1 P b5cbc627f7cd43438dd5035874204ded: No bootstrap required, opened a new log
I20260810 03:44:18.179837 11919 ts_tablet_manager.cc:1397] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:44:18.179847 11918 ts_tablet_manager.cc:1397] T 322a31f2bb60424b9acaa9faf838abc1 P b5cbc627f7cd43438dd5035874204ded: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 03:44:18.180209 11920 tablet_bootstrap.cc:654] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:18.181119 11920 log.cc:824] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:18.183775 11920 tablet_bootstrap.cc:492] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef: No bootstrap required, opened a new log
I20260810 03:44:18.183996 11920 ts_tablet_manager.cc:1397] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:44:18.184530 11918 raft_consensus.cc:348] T 322a31f2bb60424b9acaa9faf838abc1 P b5cbc627f7cd43438dd5035874204ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6eeddeb6e14a3eb7a279fa456b196e" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 32797 } } peers { permanent_uuid: "b5cbc627f7cd43438dd5035874204ded" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40963 } } peers { permanent_uuid: "a83439d0b514471091183cc3f85913ef" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 43913 } }
I20260810 03:44:18.184780 11918 raft_consensus.cc:374] T 322a31f2bb60424b9acaa9faf838abc1 P b5cbc627f7cd43438dd5035874204ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:18.184834 11918 raft_consensus.cc:729] T 322a31f2bb60424b9acaa9faf838abc1 P b5cbc627f7cd43438dd5035874204ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5cbc627f7cd43438dd5035874204ded, State: Initialized, Role: FOLLOWER
I20260810 03:44:18.184706 11919 raft_consensus.cc:348] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6eeddeb6e14a3eb7a279fa456b196e" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 32797 } } peers { permanent_uuid: "b5cbc627f7cd43438dd5035874204ded" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40963 } } peers { permanent_uuid: "a83439d0b514471091183cc3f85913ef" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 43913 } }
I20260810 03:44:18.184993 11919 raft_consensus.cc:374] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:18.185057 11919 raft_consensus.cc:729] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6eeddeb6e14a3eb7a279fa456b196e, State: Initialized, Role: FOLLOWER
I20260810 03:44:18.185351 11918 consensus_queue.cc:260] T 322a31f2bb60424b9acaa9faf838abc1 P b5cbc627f7cd43438dd5035874204ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6eeddeb6e14a3eb7a279fa456b196e" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 32797 } } peers { permanent_uuid: "b5cbc627f7cd43438dd5035874204ded" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40963 } } peers { permanent_uuid: "a83439d0b514471091183cc3f85913ef" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 43913 } }
I20260810 03:44:18.185523 11919 consensus_queue.cc:260] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6eeddeb6e14a3eb7a279fa456b196e" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 32797 } } peers { permanent_uuid: "b5cbc627f7cd43438dd5035874204ded" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40963 } } peers { permanent_uuid: "a83439d0b514471091183cc3f85913ef" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 43913 } }
I20260810 03:44:18.186249 11898 heartbeater.cc:503] Master 127.21.175.126:46597 was elected leader, sending a full tablet report...
I20260810 03:44:18.186612 11919 ts_tablet_manager.cc:1428] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 03:44:18.187307 11602 heartbeater.cc:503] Master 127.21.175.126:46597 was elected leader, sending a full tablet report...
I20260810 03:44:18.188812 11920 raft_consensus.cc:348] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6eeddeb6e14a3eb7a279fa456b196e" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 32797 } } peers { permanent_uuid: "b5cbc627f7cd43438dd5035874204ded" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40963 } } peers { permanent_uuid: "a83439d0b514471091183cc3f85913ef" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 43913 } }
I20260810 03:44:18.189054 11920 raft_consensus.cc:374] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:18.189124 11920 raft_consensus.cc:729] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a83439d0b514471091183cc3f85913ef, State: Initialized, Role: FOLLOWER
I20260810 03:44:18.189422 11918 ts_tablet_manager.cc:1428] T 322a31f2bb60424b9acaa9faf838abc1 P b5cbc627f7cd43438dd5035874204ded: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260810 03:44:18.189663 11920 consensus_queue.cc:260] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6eeddeb6e14a3eb7a279fa456b196e" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 32797 } } peers { permanent_uuid: "b5cbc627f7cd43438dd5035874204ded" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40963 } } peers { permanent_uuid: "a83439d0b514471091183cc3f85913ef" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 43913 } }
I20260810 03:44:18.190711 11752 heartbeater.cc:503] Master 127.21.175.126:46597 was elected leader, sending a full tablet report...
I20260810 03:44:18.190840 11920 ts_tablet_manager.cc:1428] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260810 03:44:18.202008 11603 tablet.cc:2369] T 322a31f2bb60424b9acaa9faf838abc1 P b5cbc627f7cd43438dd5035874204ded: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:44:18.270974 11925 raft_consensus.cc:482] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:44:18.271238 11925 raft_consensus.cc:504] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6eeddeb6e14a3eb7a279fa456b196e" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 32797 } } peers { permanent_uuid: "b5cbc627f7cd43438dd5035874204ded" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40963 } } peers { permanent_uuid: "a83439d0b514471091183cc3f85913ef" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 43913 } }
I20260810 03:44:18.272583 11925 leader_election.cc:288] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5cbc627f7cd43438dd5035874204ded (127.21.175.65:40963), a83439d0b514471091183cc3f85913ef (127.21.175.66:43913)
I20260810 03:44:18.277186 11900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:18.272911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.66:43913 (local address 127.21.175.67:34811)
0810 03:44:18.273081 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:18.273100 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:18.273218 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:18.273661 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:18.273664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:18.273685 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:18.273812 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:18.273820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.274690 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.274693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:18.275817 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:18.275827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.276754 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.276758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:18.276864 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:18.276891 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 03:44:18.276973 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 03:44:18.277045 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 03:44:18.277262 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:18.273024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:34811 (local address 127.21.175.66:43913)
0810 03:44:18.273167 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:18.273173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:18.273188 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:44:18.273364 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:18.273377 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:18.273427 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:18.273636 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:18.273933 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.274564 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.275943 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.276634 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.276786 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:18.276842 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:18.277035 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 03:44:18.277113 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:44:18.277913 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:18.273121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.175.67:59811 (local address 127.21.175.65:40963)
0810 03:44:18.273270 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:18.273276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:18.273768 (+ 492us) server_negotiation.cc:407] Connection header received
0810 03:44:18.273922 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:18.273925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:18.273978 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:18.274098 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:44:18.274375 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.275061 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.276501 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.277141 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.277283 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:18.277340 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:18.277718 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 03:44:18.277798 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:44:18.277936 11927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:18.272911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:40963 (local address 127.21.175.67:59811)
0810 03:44:18.273658 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:18.273671 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:18.273804 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:18.274109 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:18.274112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:18.274128 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:44:18.274258 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:18.274263 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.275169 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.275173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:18.276348 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:18.276360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.277328 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.277333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:18.277454 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:18.277545 (+ 91us) client_negotiation.cc:769] Sending connection context
0810 03:44:18.277691 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 03:44:18.277775 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":285,"thread_start_us":162,"threads_started":1}
I20260810 03:44:18.277886 11699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "322a31f2bb60424b9acaa9faf838abc1" candidate_uuid: "ae6eeddeb6e14a3eb7a279fa456b196e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a83439d0b514471091183cc3f85913ef" is_pre_election: true
I20260810 03:44:18.278234 11699 raft_consensus.cc:2454] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae6eeddeb6e14a3eb7a279fa456b196e in term 0.
I20260810 03:44:18.278882 11772 leader_election.cc:302] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a83439d0b514471091183cc3f85913ef, ae6eeddeb6e14a3eb7a279fa456b196e; no voters:
I20260810 03:44:18.279057 11552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "322a31f2bb60424b9acaa9faf838abc1" candidate_uuid: "ae6eeddeb6e14a3eb7a279fa456b196e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5cbc627f7cd43438dd5035874204ded" is_pre_election: true
I20260810 03:44:18.279268 11925 raft_consensus.cc:2781] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:44:18.279385 11925 raft_consensus.cc:482] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:44:18.279354 11552 raft_consensus.cc:2454] T 322a31f2bb60424b9acaa9faf838abc1 P b5cbc627f7cd43438dd5035874204ded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae6eeddeb6e14a3eb7a279fa456b196e in term 0.
I20260810 03:44:18.279448 11925 raft_consensus.cc:3037] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:18.280596 11925 raft_consensus.cc:504] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6eeddeb6e14a3eb7a279fa456b196e" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 32797 } } peers { permanent_uuid: "b5cbc627f7cd43438dd5035874204ded" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40963 } } peers { permanent_uuid: "a83439d0b514471091183cc3f85913ef" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 43913 } }
I20260810 03:44:18.280974 11925 leader_election.cc:288] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [CANDIDATE]: Term 1 election: Requested vote from peers b5cbc627f7cd43438dd5035874204ded (127.21.175.65:40963), a83439d0b514471091183cc3f85913ef (127.21.175.66:43913)
I20260810 03:44:18.281358 11552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "322a31f2bb60424b9acaa9faf838abc1" candidate_uuid: "ae6eeddeb6e14a3eb7a279fa456b196e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5cbc627f7cd43438dd5035874204ded"
I20260810 03:44:18.281401 11699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "322a31f2bb60424b9acaa9faf838abc1" candidate_uuid: "ae6eeddeb6e14a3eb7a279fa456b196e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a83439d0b514471091183cc3f85913ef"
I20260810 03:44:18.281522 11552 raft_consensus.cc:3037] T 322a31f2bb60424b9acaa9faf838abc1 P b5cbc627f7cd43438dd5035874204ded [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:18.281553 11699 raft_consensus.cc:3037] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:18.282799 11552 raft_consensus.cc:2454] T 322a31f2bb60424b9acaa9faf838abc1 P b5cbc627f7cd43438dd5035874204ded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae6eeddeb6e14a3eb7a279fa456b196e in term 1.
I20260810 03:44:18.283147 11699 raft_consensus.cc:2454] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae6eeddeb6e14a3eb7a279fa456b196e in term 1.
I20260810 03:44:18.283195 11773 leader_election.cc:302] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae6eeddeb6e14a3eb7a279fa456b196e, b5cbc627f7cd43438dd5035874204ded; no voters:
I20260810 03:44:18.283386 11925 raft_consensus.cc:2781] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:18.283639 11925 raft_consensus.cc:686] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [term 1 LEADER]: Becoming Leader. State: Replica: ae6eeddeb6e14a3eb7a279fa456b196e, State: Running, Role: LEADER
I20260810 03:44:18.283993 11925 consensus_queue.cc:237] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6eeddeb6e14a3eb7a279fa456b196e" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 32797 } } peers { permanent_uuid: "b5cbc627f7cd43438dd5035874204ded" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40963 } } peers { permanent_uuid: "a83439d0b514471091183cc3f85913ef" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 43913 } }
I20260810 03:44:18.287214 11391 catalog_manager.cc:5495] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e reported cstate change: term changed from 0 to 1, leader changed from <none> to ae6eeddeb6e14a3eb7a279fa456b196e (127.21.175.67). New cstate: current_term: 1 leader_uuid: "ae6eeddeb6e14a3eb7a279fa456b196e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6eeddeb6e14a3eb7a279fa456b196e" member_type: VOTER last_known_addr { host: "127.21.175.67" port: 32797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5cbc627f7cd43438dd5035874204ded" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a83439d0b514471091183cc3f85913ef" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 43913 } health_report { overall_health: UNKNOWN } } }
I20260810 03:44:18.314771 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:18.310920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45664 (local address 127.21.175.67:32797)
0810 03:44:18.311054 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:18.311058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:18.311071 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:44:18.311239 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:18.311242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:18.311301 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:18.311398 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:44:18.311644 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.312188 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.313280 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:18.313522 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:18.313606 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:18.313904 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:44:18.314358 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:44:18.314453 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:18.314604 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 03:44:18.314658 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:44:18.321169 11699 raft_consensus.cc:1261] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef [term 1 FOLLOWER]: Refusing update from remote peer ae6eeddeb6e14a3eb7a279fa456b196e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:44:18.321170 11552 raft_consensus.cc:1261] T 322a31f2bb60424b9acaa9faf838abc1 P b5cbc627f7cd43438dd5035874204ded [term 1 FOLLOWER]: Refusing update from remote peer ae6eeddeb6e14a3eb7a279fa456b196e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:44:18.322010 11925 consensus_queue.cc:1035] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a83439d0b514471091183cc3f85913ef" member_type: VOTER last_known_addr { host: "127.21.175.66" port: 43913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:44:18.322208 11928 consensus_queue.cc:1035] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5cbc627f7cd43438dd5035874204ded" member_type: VOTER last_known_addr { host: "127.21.175.65" port: 40963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:44:18.340029 11391 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:44:18.340245 11391 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:44:18.343152 11391 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0eacbc4522fe4a46b132523a58eef3fe: Sending DeleteTablet for 3 replicas of tablet 322a31f2bb60424b9acaa9faf838abc1
I20260810 03:44:18.344012 11676 tablet_service.cc:1514] Processing DeleteTablet for tablet 322a31f2bb60424b9acaa9faf838abc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:18 UTC) from {username='slave'} at 127.0.0.1:58642
I20260810 03:44:18.344012 11528 tablet_service.cc:1514] Processing DeleteTablet for tablet 322a31f2bb60424b9acaa9faf838abc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:18 UTC) from {username='slave'} at 127.0.0.1:35330
I20260810 03:44:18.344049 11823 tablet_service.cc:1514] Processing DeleteTablet for tablet 322a31f2bb60424b9acaa9faf838abc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:44:18 UTC) from {username='slave'} at 127.0.0.1:45652
I20260810 03:44:18.344635 11943 tablet_replica.cc:330] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e: stopping tablet replica
I20260810 03:44:18.344987 11943 raft_consensus.cc:2229] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:44:18.345642 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 11456
I20260810 03:44:18.345722 11941 tablet_replica.cc:330] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef: stopping tablet replica
I20260810 03:44:18.346028 11941 raft_consensus.cc:2229] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:44:18.346879 11941 raft_consensus.cc:2258] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:44:18.345443 11943 raft_consensus.cc:2258] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:44:18.349859 11941 ts_tablet_manager.cc:1905] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:44:18.353235 11941 ts_tablet_manager.cc:1918] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:44:18.353335 11941 log.cc:1196] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-1/wal/wals/322a31f2bb60424b9acaa9faf838abc1
I20260810 03:44:18.353667 11941 ts_tablet_manager.cc:1939] T 322a31f2bb60424b9acaa9faf838abc1 P a83439d0b514471091183cc3f85913ef: Deleting consensus metadata
I20260810 03:44:18.355187 11372 catalog_manager.cc:4841] TS a83439d0b514471091183cc3f85913ef (127.21.175.66:43913): tablet 322a31f2bb60424b9acaa9faf838abc1 (table test-table [id=a3c00f56708c4b099e685ddd5637b7a0]) successfully deleted
I20260810 03:44:18.359246 11943 ts_tablet_manager.cc:1905] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:44:18.363056 11943 ts_tablet_manager.cc:1918] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:44:18.363175 11943 log.cc:1196] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e: Deleting WAL directory at /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333190020696-22205-0/minicluster-data/ts-2/wal/wals/322a31f2bb60424b9acaa9faf838abc1
W20260810 03:44:18.363304 11373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.175.65:40963 (error 108)
I20260810 03:44:18.363513 11943 ts_tablet_manager.cc:1939] T 322a31f2bb60424b9acaa9faf838abc1 P ae6eeddeb6e14a3eb7a279fa456b196e: Deleting consensus metadata
I20260810 03:44:18.364511 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:18.364171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.65:40963 (local address 127.0.0.1:35332)
0810 03:44:18.364324 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:18.364403 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.65:40963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 03:44:18.364578 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 11605
W20260810 03:44:18.364832 11373 catalog_manager.cc:4584] TS b5cbc627f7cd43438dd5035874204ded (127.21.175.65:40963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 322a31f2bb60424b9acaa9faf838abc1: Network error: Client connection negotiation failed: client connection to 127.21.175.65:40963: connect: Connection refused (error 111)
I20260810 03:44:18.366835 11374 catalog_manager.cc:4841] TS ae6eeddeb6e14a3eb7a279fa456b196e (127.21.175.67:32797): tablet 322a31f2bb60424b9acaa9faf838abc1 (table test-table [id=a3c00f56708c4b099e685ddd5637b7a0]) successfully deleted
I20260810 03:44:18.374634 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 11755
I20260810 03:44:18.382846 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 11353
2026-08-10T03:44:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1193 ms)
[----------] 6 tests from EncryptionPolicyTest (11496 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:18 dist-test-slave-ss4n krb5kdc[11947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:18 dist-test-slave-ss4n krb5kdc[11947](info): set up 2 sockets
Aug 10 03:44:18 dist-test-slave-ss4n krb5kdc[11947](info): commencing operation
krb5kdc: starting...
W20260810 03:44:19.488917 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:19 dist-test-slave-ss4n krb5kdc[11947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:44:19.628674 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:19.752650 11961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:19.753037 11961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:19.753156 11961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:19.757848 11961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:19.757942 11961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:19.757984 11961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:19.758047 11961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:19.758090 11961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:19.764281 11961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41269
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:19.765616 11961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:19.766778 11961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:19.776625 11968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:19.776870 11961 server_base.cc:1034] running on GCE node
W20260810 03:44:19.776625 11967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:19.776729 11971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:19.777726 11961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:19.778905 11961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:19.780112 11961 hybrid_clock.cc:648] HybridClock initialized: now 1786333459780096 us; error 31 us; skew 500 ppm
Aug 10 03:44:19 dist-test-slave-ss4n krb5kdc[11947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:19.786947 11961 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:44:19.788023 11961 webserver.cc:460] Webserver started at http://127.0.0.1:37987/ using document root <none> and password file <none>
I20260810 03:44:19.788674 11961 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:19.788791 11961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:19.789063 11961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:19.791487 11961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "e588649122f241e3a19d2ddd95c48615"
format_stamp: "Formatted at 2026-08-10 03:44:19 on dist-test-slave-ss4n"
server_key: "4147f36b8e28c7141a5112276107b709"
server_key_iv: "99468f04c6f7e7f8d7003b697287c6c5"
server_key_version: "encryptionkey@0"
I20260810 03:44:19.792172 11961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "e588649122f241e3a19d2ddd95c48615"
format_stamp: "Formatted at 2026-08-10 03:44:19 on dist-test-slave-ss4n"
server_key: "4147f36b8e28c7141a5112276107b709"
server_key_iv: "99468f04c6f7e7f8d7003b697287c6c5"
server_key_version: "encryptionkey@0"
I20260810 03:44:19.796528 11961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:44:19.799314 11978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:19.800666 11961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:44:19.800833 11961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "e588649122f241e3a19d2ddd95c48615"
format_stamp: "Formatted at 2026-08-10 03:44:19 on dist-test-slave-ss4n"
server_key: "4147f36b8e28c7141a5112276107b709"
server_key_iv: "99468f04c6f7e7f8d7003b697287c6c5"
server_key_version: "encryptionkey@0"
I20260810 03:44:19.800969 11961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:19.825322 11961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:19.829200 11961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:19.829440 11961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:19.837373 11961 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41269
I20260810 03:44:19.837390 12035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41269 every 8 connection(s)
I20260810 03:44:19.838513 11961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:19.841647 12038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:19.846944 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 11961
I20260810 03:44:19.847116 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:19.847438 22205 external_mini_cluster.cc:1428] Setting key 6b6dd941a402ed3e307b380d4b2d9d23
I20260810 03:44:19.847761 12038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615: Bootstrap starting.
I20260810 03:44:19.850517 12038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:19.851451 12038 log.cc:824] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:19.853775 12038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615: No bootstrap required, opened a new log
Aug 10 03:44:19 dist-test-slave-ss4n krb5kdc[11947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 03:44:19.858587 12038 raft_consensus.cc:348] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e588649122f241e3a19d2ddd95c48615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41269 } }
I20260810 03:44:19.858872 12038 raft_consensus.cc:374] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:19.858938 12038 raft_consensus.cc:729] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e588649122f241e3a19d2ddd95c48615, State: Initialized, Role: FOLLOWER
I20260810 03:44:19.859450 12038 consensus_queue.cc:260] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e588649122f241e3a19d2ddd95c48615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41269 } }
I20260810 03:44:19.859578 12038 raft_consensus.cc:388] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:19.859663 12038 raft_consensus.cc:482] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:19.859789 12038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:19.860754 12038 raft_consensus.cc:504] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e588649122f241e3a19d2ddd95c48615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41269 } }
I20260810 03:44:19.861122 12038 leader_election.cc:302] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e588649122f241e3a19d2ddd95c48615; no voters:
I20260810 03:44:19.861552 12038 leader_election.cc:288] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:19.861855 12044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:19.862090 12044 raft_consensus.cc:686] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [term 1 LEADER]: Becoming Leader. State: Replica: e588649122f241e3a19d2ddd95c48615, State: Running, Role: LEADER
I20260810 03:44:19.862501 12044 consensus_queue.cc:237] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e588649122f241e3a19d2ddd95c48615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41269 } }
I20260810 03:44:19.863006 12038 sys_catalog.cc:564] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:19.864936 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:19.849043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47837 (local address 127.0.0.1:41269)
0810 03:44:19.849559 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:19.849586 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:19.849627 (+ 41us) server_negotiation.cc:407] Connection header received
0810 03:44:19.850346 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:19.850363 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:19.850807 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:19.851130 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:19.852114 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:19.853406 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:19.854698 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:19.855226 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:19.855360 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:19.858734 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:19.858739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:19.858755 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:19.858787 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:19.861195 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:19.861946 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:19.861953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:19.861961 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:19.862132 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:19.862553 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:19.862556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:19.862558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:19.863095 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:19.863322 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:19.863729 (+ 407us) server_negotiation.cc:299] Negotiation successful
0810 03:44:19.863974 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":311,"thread_start_us":165,"threads_started":1}
I20260810 03:44:19.866494 12045 sys_catalog.cc:455] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e588649122f241e3a19d2ddd95c48615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e588649122f241e3a19d2ddd95c48615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41269 } } }
I20260810 03:44:19.866628 12045 sys_catalog.cc:458] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:19.866941 12047 sys_catalog.cc:455] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e588649122f241e3a19d2ddd95c48615. Latest consensus state: current_term: 1 leader_uuid: "e588649122f241e3a19d2ddd95c48615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e588649122f241e3a19d2ddd95c48615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41269 } } }
I20260810 03:44:19.867030 12047 sys_catalog.cc:458] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:19.868561 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 03:44:19.868615 12060 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:44:19.868700 12060 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:44:19.868773 12063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:19.869473 12063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:19.875504 12063 catalog_manager.cc:1347] Generated new cluster ID: 407a4e3eb8394751a04ef5caf17f1afb
I20260810 03:44:19.875582 12063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:19.879508 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:19.870811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57993 (local address 127.0.0.1:41269)
0810 03:44:19.870950 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:19.870956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:19.871185 (+ 229us) server_negotiation.cc:407] Connection header received
0810 03:44:19.871345 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:19.871348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:19.871399 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:19.871480 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:19.872193 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:19.872725 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:19.873754 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:19.874067 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:19.874186 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:19.876044 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:19.876047 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:19.876049 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:19.876072 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:19.877693 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:19.878225 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:19.878228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:19.878229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:19.878306 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:19.878616 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:19.878619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:19.878620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:19.878827 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:19.878941 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:19.879215 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 03:44:19.879329 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:44:19.905989 12063 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:19.906978 12063 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:19.915820 12063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e588649122f241e3a19d2ddd95c48615: Generated new TSK 0
I20260810 03:44:19.917456 12063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:19.922787 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 11961
W20260810 03:44:19.932152 11945 connection.cc:535] client connection to 127.0.0.1:41269 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41269): Connection reset by peer (error 104)
2026-08-10T03:44:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1538 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:19 dist-test-slave-ss4n krb5kdc[12073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:19 dist-test-slave-ss4n krb5kdc[12073](info): set up 2 sockets
Aug 10 03:44:19 dist-test-slave-ss4n krb5kdc[12073](info): commencing operation
krb5kdc: starting...
W20260810 03:44:21.046496 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:21 dist-test-slave-ss4n krb5kdc[12073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:44:21.188871 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:21.304643 12087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:21.304935 12087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:21.304977 12087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:21.309011 12087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:21.309087 12087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:21.309106 12087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:21.309122 12087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:21.309139 12087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:21.314647 12087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36477
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:21.315785 12087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:21.316916 12087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:21.326264 12092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:21.326264 12093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:21.326357 12087 server_base.cc:1034] running on GCE node
W20260810 03:44:21.326295 12095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:21.327112 12087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:21.328435 12087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:21.329584 12087 hybrid_clock.cc:648] HybridClock initialized: now 1786333461329553 us; error 47 us; skew 500 ppm
Aug 10 03:44:21 dist-test-slave-ss4n krb5kdc[12073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:21.332935 12087 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:44:21.334152 12087 webserver.cc:460] Webserver started at http://127.0.0.1:42873/ using document root <none> and password file <none>
I20260810 03:44:21.334791 12087 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:21.334865 12087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:21.335111 12087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:21.337420 12087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "01681236a433451c96bdd676020f40dc"
format_stamp: "Formatted at 2026-08-10 03:44:21 on dist-test-slave-ss4n"
server_key: "a3c19ae18d594e144c86eae82425c891"
server_key_iv: "ea58d595c6f1fec307bac9d32e46c9e4"
server_key_version: "encryptionkey@0"
I20260810 03:44:21.338161 12087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "01681236a433451c96bdd676020f40dc"
format_stamp: "Formatted at 2026-08-10 03:44:21 on dist-test-slave-ss4n"
server_key: "a3c19ae18d594e144c86eae82425c891"
server_key_iv: "ea58d595c6f1fec307bac9d32e46c9e4"
server_key_version: "encryptionkey@0"
I20260810 03:44:21.342502 12087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:44:21.345585 12103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:21.347185 12087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:44:21.347313 12087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "01681236a433451c96bdd676020f40dc"
format_stamp: "Formatted at 2026-08-10 03:44:21 on dist-test-slave-ss4n"
server_key: "a3c19ae18d594e144c86eae82425c891"
server_key_iv: "ea58d595c6f1fec307bac9d32e46c9e4"
server_key_version: "encryptionkey@0"
I20260810 03:44:21.347422 12087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:21.370217 12087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:21.373849 12087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:21.374060 12087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:21.382814 12087 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36477
I20260810 03:44:21.382843 12160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36477 every 8 connection(s)
I20260810 03:44:21.383960 12087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:21.386739 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 12087
I20260810 03:44:21.386925 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:21.387034 12161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:21.387310 22205 external_mini_cluster.cc:1428] Setting key 89ebb0cba773643e66acc0c20e0fe2bb
I20260810 03:44:21.393980 12161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc: Bootstrap starting.
I20260810 03:44:21.396611 12161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:21.397393 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:21.388865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50281 (local address 127.0.0.1:36477)
0810 03:44:21.389322 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:21.389339 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:21.389377 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:44:21.390133 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:21.390153 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:21.390501 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:21.390801 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:21.391154 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:21.392158 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:21.393412 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:21.394108 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:21.394320 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:21.395270 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:21.395276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:21.395290 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:21.395751 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:21.395815 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:21.395844 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:21.395984 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:21.396289 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 03:44:21.396525 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":153,"threads_started":1}
I20260810 03:44:21.397401 12161 log.cc:824] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:21.399593 12161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc: No bootstrap required, opened a new log
I20260810 03:44:21.400833 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:44:21.404377 12161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01681236a433451c96bdd676020f40dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36477 } }
I20260810 03:44:21.404629 12161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:21.404690 12161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01681236a433451c96bdd676020f40dc, State: Initialized, Role: FOLLOWER
I20260810 03:44:21.405174 12161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01681236a433451c96bdd676020f40dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36477 } }
I20260810 03:44:21.405303 12161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:21.405385 12161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:21.405525 12161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:21.406497 12161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01681236a433451c96bdd676020f40dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36477 } }
I20260810 03:44:21.406857 12161 leader_election.cc:302] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01681236a433451c96bdd676020f40dc; no voters:
I20260810 03:44:21.407226 12161 leader_election.cc:288] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:21.407297 12171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:21.407516 12171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [term 1 LEADER]: Becoming Leader. State: Replica: 01681236a433451c96bdd676020f40dc, State: Running, Role: LEADER
I20260810 03:44:21.407826 12171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01681236a433451c96bdd676020f40dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36477 } }
I20260810 03:44:21.408037 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:21.403311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58619 (local address 127.0.0.1:36477)
0810 03:44:21.403528 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:21.403533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:21.403670 (+ 137us) server_negotiation.cc:407] Connection header received
0810 03:44:21.403800 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:21.403803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:21.403845 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:21.403917 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:21.404219 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:21.404898 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:21.406073 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:21.406346 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:21.406484 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:21.407515 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:21.407520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:21.407525 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:21.407543 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:21.407574 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:21.407591 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:21.407700 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:21.407844 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 03:44:21.407880 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:44:21.408604 12161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:21.409042 12171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01681236a433451c96bdd676020f40dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01681236a433451c96bdd676020f40dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36477 } } }
I20260810 03:44:21.409157 12171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:21.409889 12173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01681236a433451c96bdd676020f40dc. Latest consensus state: current_term: 1 leader_uuid: "01681236a433451c96bdd676020f40dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01681236a433451c96bdd676020f40dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36477 } } }
I20260810 03:44:21.410037 12173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:21.410323 12175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:21.414562 12175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:21.421375 12175 catalog_manager.cc:1347] Generated new cluster ID: b06ae45761cf46a4a17716d96d465fbb
I20260810 03:44:21.421469 12175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:21.428354 12175 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:21.429703 12175 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:21.434589 12175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01681236a433451c96bdd676020f40dc: Generated new TSK 0
I20260810 03:44:21.435424 12175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:21.440193 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 12087
W20260810 03:44:21.450021 12071 connection.cc:535] client connection to 127.0.0.1:36477 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36477): Connection reset by peer (error 104)
2026-08-10T03:44:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1517 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:21 dist-test-slave-ss4n krb5kdc[12199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:21 dist-test-slave-ss4n krb5kdc[12199](info): set up 2 sockets
Aug 10 03:44:21 dist-test-slave-ss4n krb5kdc[12199](info): commencing operation
krb5kdc: starting...
W20260810 03:44:22.532816 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:22 dist-test-slave-ss4n krb5kdc[12199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:44:22.669147 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33623
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:22.789080 12212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:22.789450 12212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:22.789570 12212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:22.793702 12212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:22.793812 12212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:22.793860 12212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:22.793921 12212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:22.793962 12212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:22.799510 12212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33623
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:22.800709 12212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:22.801852 12212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:22.811175 12221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:22.811169 12219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:22.811319 12212 server_base.cc:1034] running on GCE node
W20260810 03:44:22.811169 12218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:22.812249 12212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:22.813638 12212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:22.814862 12212 hybrid_clock.cc:648] HybridClock initialized: now 1786333462814841 us; error 32 us; skew 500 ppm
Aug 10 03:44:22 dist-test-slave-ss4n krb5kdc[12199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:22.817786 12212 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:44:22.818806 12212 webserver.cc:460] Webserver started at http://127.0.0.1:45857/ using document root <none> and password file <none>
I20260810 03:44:22.819355 12212 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:22.819414 12212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:22.819588 12212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:22.821767 12212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "a703600a376e4378bdd66f16a30743f2"
format_stamp: "Formatted at 2026-08-10 03:44:22 on dist-test-slave-ss4n"
server_key: "39971e72c1690697d6bb4d8d8fd0ab12"
server_key_iv: "236b217e5602461f1eca6202fdad09ee"
server_key_version: "encryptionkey@0"
I20260810 03:44:22.822340 12212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "a703600a376e4378bdd66f16a30743f2"
format_stamp: "Formatted at 2026-08-10 03:44:22 on dist-test-slave-ss4n"
server_key: "39971e72c1690697d6bb4d8d8fd0ab12"
server_key_iv: "236b217e5602461f1eca6202fdad09ee"
server_key_version: "encryptionkey@0"
I20260810 03:44:22.826393 12212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:44:22.828881 12229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:22.830243 12212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 03:44:22.830379 12212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "a703600a376e4378bdd66f16a30743f2"
format_stamp: "Formatted at 2026-08-10 03:44:22 on dist-test-slave-ss4n"
server_key: "39971e72c1690697d6bb4d8d8fd0ab12"
server_key_iv: "236b217e5602461f1eca6202fdad09ee"
server_key_version: "encryptionkey@0"
I20260810 03:44:22.830492 12212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:22.835265 12212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:22.838382 12212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:22.838586 12212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:22.846206 12212 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33623
I20260810 03:44:22.846237 12284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33623 every 8 connection(s)
I20260810 03:44:22.847246 12212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:22.850382 12285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:22.856217 12285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2: Bootstrap starting.
I20260810 03:44:22.856734 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 12212
I20260810 03:44:22.856915 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:22.857208 22205 external_mini_cluster.cc:1428] Setting key 13bd3458eb432cbdfc9167a7a5fa8138
I20260810 03:44:22.859241 12285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:22.860157 12285 log.cc:824] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:22.862601 12285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2: No bootstrap required, opened a new log
I20260810 03:44:22.863322 12288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:22.858755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57457 (local address 127.0.0.1:33623)
0810 03:44:22.859266 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:22.859286 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:22.859332 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:44:22.860256 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:22.860282 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:22.860668 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:22.861005 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:22.861864 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:22.861869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:22.861884 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:22.861897 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:22.861965 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:22.861992 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:22.862109 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:22.862199 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 03:44:22.862356 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":143,"threads_started":1}
I20260810 03:44:22.866298 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:44:22.867146 12285 raft_consensus.cc:348] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a703600a376e4378bdd66f16a30743f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33623 } }
I20260810 03:44:22.867399 12285 raft_consensus.cc:374] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:22.867460 12285 raft_consensus.cc:729] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a703600a376e4378bdd66f16a30743f2, State: Initialized, Role: FOLLOWER
I20260810 03:44:22.867921 12285 consensus_queue.cc:260] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a703600a376e4378bdd66f16a30743f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33623 } }
I20260810 03:44:22.868076 12285 raft_consensus.cc:388] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:22.868170 12285 raft_consensus.cc:482] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:22.868273 12285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:22.869530 12285 raft_consensus.cc:504] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a703600a376e4378bdd66f16a30743f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33623 } }
I20260810 03:44:22.869913 12285 leader_election.cc:302] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a703600a376e4378bdd66f16a30743f2; no voters:
I20260810 03:44:22.870276 12285 leader_election.cc:288] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:22.870417 12295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:22.870458 12288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:22.868610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47519 (local address 127.0.0.1:33623)
0810 03:44:22.868730 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:22.868735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:22.868969 (+ 234us) server_negotiation.cc:407] Connection header received
0810 03:44:22.869045 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:22.869048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:22.869085 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:22.869169 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:22.869939 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:22.869943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:22.869945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:22.869957 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:22.869985 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:22.869997 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:22.870086 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:22.870280 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 03:44:22.870326 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 03:44:22.870744 12295 raft_consensus.cc:686] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [term 1 LEADER]: Becoming Leader. State: Replica: a703600a376e4378bdd66f16a30743f2, State: Running, Role: LEADER
I20260810 03:44:22.871071 12295 consensus_queue.cc:237] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a703600a376e4378bdd66f16a30743f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33623 } }
I20260810 03:44:22.871680 12285 sys_catalog.cc:564] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:22.872372 12295 sys_catalog.cc:455] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a703600a376e4378bdd66f16a30743f2. Latest consensus state: current_term: 1 leader_uuid: "a703600a376e4378bdd66f16a30743f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a703600a376e4378bdd66f16a30743f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33623 } } }
I20260810 03:44:22.872495 12295 sys_catalog.cc:458] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:22.872983 12297 sys_catalog.cc:455] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a703600a376e4378bdd66f16a30743f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a703600a376e4378bdd66f16a30743f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33623 } } }
I20260810 03:44:22.873143 12297 sys_catalog.cc:458] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:22.874017 12303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:22.876777 12303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:22.881976 12303 catalog_manager.cc:1347] Generated new cluster ID: ceeb80ebf2334278bd643c0511ab2ded
I20260810 03:44:22.882048 12303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:22.891573 12303 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:22.891925 12303 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:22.896816 12303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a703600a376e4378bdd66f16a30743f2: Generated new TSK 0
I20260810 03:44:22.897576 12303 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:22.904269 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 12212
2026-08-10T03:44:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1462 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:22 dist-test-slave-ss4n krb5kdc[12318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:22 dist-test-slave-ss4n krb5kdc[12318](info): set up 2 sockets
Aug 10 03:44:22 dist-test-slave-ss4n krb5kdc[12318](info): commencing operation
krb5kdc: starting...
W20260810 03:44:24.077087 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:24 dist-test-slave-ss4n krb5kdc[12318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:44:24.215107 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37187
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:24.335179 12333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:24.335618 12333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:24.335716 12333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:24.339993 12333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:24.340104 12333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:24.340142 12333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:24.340184 12333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:24.340216 12333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:24.345881 12333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:24.347095 12333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:24.348212 12333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:24.357384 12339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:24.357406 12340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:24.357754 12333 server_base.cc:1034] running on GCE node
W20260810 03:44:24.357406 12343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:24.358470 12333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:24.359700 12333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:24.360927 12333 hybrid_clock.cc:648] HybridClock initialized: now 1786333464360890 us; error 49 us; skew 500 ppm
Aug 10 03:44:24 dist-test-slave-ss4n krb5kdc[12318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:24.363915 12333 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:44:24.364866 12333 webserver.cc:460] Webserver started at http://127.0.0.1:40219/ using document root <none> and password file <none>
I20260810 03:44:24.365449 12333 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:24.365651 12333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:24.365912 12333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:24.368201 12333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "cb50fa64e3e24f66b7a2e726cb87743a"
format_stamp: "Formatted at 2026-08-10 03:44:24 on dist-test-slave-ss4n"
server_key: "24ca0431fb7df0545972224dd57363a9"
server_key_iv: "924ac819529e31c69201791b73e3f04a"
server_key_version: "encryptionkey@0"
I20260810 03:44:24.368856 12333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "cb50fa64e3e24f66b7a2e726cb87743a"
format_stamp: "Formatted at 2026-08-10 03:44:24 on dist-test-slave-ss4n"
server_key: "24ca0431fb7df0545972224dd57363a9"
server_key_iv: "924ac819529e31c69201791b73e3f04a"
server_key_version: "encryptionkey@0"
I20260810 03:44:24.372973 12333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260810 03:44:24.375670 12351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:24.376981 12333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:44:24.377134 12333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "cb50fa64e3e24f66b7a2e726cb87743a"
format_stamp: "Formatted at 2026-08-10 03:44:24 on dist-test-slave-ss4n"
server_key: "24ca0431fb7df0545972224dd57363a9"
server_key_iv: "924ac819529e31c69201791b73e3f04a"
server_key_version: "encryptionkey@0"
I20260810 03:44:24.377259 12333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:24.393920 12333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:24.397392 12333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:24.397704 12333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:24.405126 12333 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37187
I20260810 03:44:24.405156 12408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37187 every 8 connection(s)
I20260810 03:44:24.406131 12333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:24.408937 12409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:24.413157 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 12333
I20260810 03:44:24.413282 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:24.413681 22205 external_mini_cluster.cc:1428] Setting key 0ee02e1bd157da7e73580867ff594983
I20260810 03:44:24.414896 12409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a: Bootstrap starting.
I20260810 03:44:24.417722 12409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:24.418596 12409 log.cc:824] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:24.420802 12409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a: No bootstrap required, opened a new log
Aug 10 03:44:24 dist-test-slave-ss4n krb5kdc[12318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 03:44:24.425202 12409 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb50fa64e3e24f66b7a2e726cb87743a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37187 } }
I20260810 03:44:24.425549 12409 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:24.425654 12409 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb50fa64e3e24f66b7a2e726cb87743a, State: Initialized, Role: FOLLOWER
I20260810 03:44:24.426244 12409 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb50fa64e3e24f66b7a2e726cb87743a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37187 } }
I20260810 03:44:24.426435 12409 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:24.426563 12409 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:24.426697 12409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:24.427917 12409 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb50fa64e3e24f66b7a2e726cb87743a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37187 } }
I20260810 03:44:24.428314 12409 leader_election.cc:302] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb50fa64e3e24f66b7a2e726cb87743a; no voters:
I20260810 03:44:24.428747 12409 leader_election.cc:288] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:24.428859 12414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:24.429095 12414 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [term 1 LEADER]: Becoming Leader. State: Replica: cb50fa64e3e24f66b7a2e726cb87743a, State: Running, Role: LEADER
I20260810 03:44:24.429471 12414 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb50fa64e3e24f66b7a2e726cb87743a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37187 } }
I20260810 03:44:24.430299 12409 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:24.430774 12414 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb50fa64e3e24f66b7a2e726cb87743a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb50fa64e3e24f66b7a2e726cb87743a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37187 } } }
I20260810 03:44:24.430948 12414 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:24.431255 12415 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb50fa64e3e24f66b7a2e726cb87743a. Latest consensus state: current_term: 1 leader_uuid: "cb50fa64e3e24f66b7a2e726cb87743a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb50fa64e3e24f66b7a2e726cb87743a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37187 } } }
I20260810 03:44:24.431375 12415 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:24.431780 12421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:24.432022 12412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:24.415225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34199 (local address 127.0.0.1:37187)
0810 03:44:24.415689 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:24.415708 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:24.415752 (+ 44us) server_negotiation.cc:407] Connection header received
0810 03:44:24.416705 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:24.416732 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:24.417182 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:24.417597 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:24.418459 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:24.419829 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:24.420798 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:24.421533 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:24.421699 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:24.424789 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:24.424796 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:24.424816 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:24.424853 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:24.427312 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:24.427865 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:24.427873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:24.427879 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:24.428041 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:24.428397 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:24.428400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:24.428402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:24.429159 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:24.430185 (+ 1026us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:24.430359 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 03:44:24.430761 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":137,"threads_started":1}
I20260810 03:44:24.435267 12421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:24.436134 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:44:24.442655 12421 catalog_manager.cc:1347] Generated new cluster ID: 83da856f05ae497bbe406fbd79b6a31a
I20260810 03:44:24.442749 12421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:24.448827 12412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:24.438756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34375 (local address 127.0.0.1:37187)
0810 03:44:24.439787 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:24.439793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:24.439809 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:44:24.439851 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:24.439856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:24.439909 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:24.440022 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:24.440900 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:24.441610 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:24.442822 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:24.443210 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:24.443336 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:24.445093 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:24.445097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:24.445100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:24.445129 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:24.446987 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:24.447502 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:24.447505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:24.447507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:24.447600 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:24.447903 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:24.447906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:24.447908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:24.448122 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:24.448247 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:24.448527 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 03:44:24.448652 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":952}
I20260810 03:44:24.453594 12421 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:24.454605 12421 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:24.464725 12421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb50fa64e3e24f66b7a2e726cb87743a: Generated new TSK 0
I20260810 03:44:24.465603 12421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:24.471276 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 12333
2026-08-10T03:44:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1567 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:24 dist-test-slave-ss4n krb5kdc[12445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:24 dist-test-slave-ss4n krb5kdc[12445](info): set up 2 sockets
Aug 10 03:44:24 dist-test-slave-ss4n krb5kdc[12445](info): commencing operation
krb5kdc: starting...
W20260810 03:44:25.651296 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:25 dist-test-slave-ss4n krb5kdc[12445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:44:25.792927 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:25.909201 12459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:25.909603 12459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:25.909720 12459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:25.914212 12459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:25.914322 12459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:25.914371 12459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:25.914448 12459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:25.914482 12459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:25.922451 12459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:25.923666 12459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:25.924762 12459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:25.934042 12465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:25.934134 12459 server_base.cc:1034] running on GCE node
W20260810 03:44:25.934099 12464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:25.934276 12467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:25.934834 12459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:25.936053 12459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:25.937264 12459 hybrid_clock.cc:648] HybridClock initialized: now 1786333465937242 us; error 32 us; skew 500 ppm
Aug 10 03:44:25 dist-test-slave-ss4n krb5kdc[12445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:25.945362 12459 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:44:25.946619 12459 webserver.cc:460] Webserver started at http://127.0.0.1:42103/ using document root <none> and password file <none>
I20260810 03:44:25.947296 12459 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:25.947408 12459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:25.947690 12459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:25.950255 12459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "330c48de0d714564b0a7c17e3f69028c"
format_stamp: "Formatted at 2026-08-10 03:44:25 on dist-test-slave-ss4n"
server_key: "94a54e43b44de5beca9ea79a2b4030f1"
server_key_iv: "12ad5f71d94e1c115208582576e4ba9b"
server_key_version: "encryptionkey@0"
I20260810 03:44:25.950939 12459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "330c48de0d714564b0a7c17e3f69028c"
format_stamp: "Formatted at 2026-08-10 03:44:25 on dist-test-slave-ss4n"
server_key: "94a54e43b44de5beca9ea79a2b4030f1"
server_key_iv: "12ad5f71d94e1c115208582576e4ba9b"
server_key_version: "encryptionkey@0"
I20260810 03:44:25.955317 12459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:44:25.958285 12476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:25.959810 12459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:44:25.960014 12459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "330c48de0d714564b0a7c17e3f69028c"
format_stamp: "Formatted at 2026-08-10 03:44:25 on dist-test-slave-ss4n"
server_key: "94a54e43b44de5beca9ea79a2b4030f1"
server_key_iv: "12ad5f71d94e1c115208582576e4ba9b"
server_key_version: "encryptionkey@0"
I20260810 03:44:25.960176 12459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:25.983256 12459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:25.986663 12459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:25.986902 12459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:25.995417 12459 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42705
I20260810 03:44:25.995476 12537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42705 every 8 connection(s)
I20260810 03:44:25.996546 12459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:25.999500 12538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:26.001183 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 12459
I20260810 03:44:26.001362 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:26.001693 22205 external_mini_cluster.cc:1428] Setting key be8f64699e67cf94e0b48db0016a1adb
I20260810 03:44:26.006495 12538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c: Bootstrap starting.
I20260810 03:44:26.009305 12538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:26.010284 12538 log.cc:824] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:26.012117 12541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:26.003483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42163 (local address 127.0.0.1:42705)
0810 03:44:26.003948 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:26.003973 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:26.004012 (+ 39us) server_negotiation.cc:407] Connection header received
0810 03:44:26.004723 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:26.004742 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:26.005113 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:26.005413 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:26.005849 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:26.007018 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:26.008290 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:26.008832 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:26.008964 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:26.009982 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:26.009988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:26.010003 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:26.010542 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:26.010658 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:26.010687 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:26.010817 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:26.011039 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 03:44:26.011296 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":149,"threads_started":1}
I20260810 03:44:26.012462 12538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c: No bootstrap required, opened a new log
I20260810 03:44:26.015652 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:44:26.016613 12538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330c48de0d714564b0a7c17e3f69028c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } }
I20260810 03:44:26.016880 12538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:26.017000 12538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 330c48de0d714564b0a7c17e3f69028c, State: Initialized, Role: FOLLOWER
I20260810 03:44:26.017444 12538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330c48de0d714564b0a7c17e3f69028c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } }
I20260810 03:44:26.017652 12538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:26.017727 12538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:26.017832 12538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:26.019176 12538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330c48de0d714564b0a7c17e3f69028c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } }
I20260810 03:44:26.019567 12538 leader_election.cc:302] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 330c48de0d714564b0a7c17e3f69028c; no voters:
I20260810 03:44:26.019939 12538 leader_election.cc:288] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:26.020066 12549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:26.020327 12549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [term 1 LEADER]: Becoming Leader. State: Replica: 330c48de0d714564b0a7c17e3f69028c, State: Running, Role: LEADER
I20260810 03:44:26.020690 12549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330c48de0d714564b0a7c17e3f69028c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } }
I20260810 03:44:26.021339 12538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:26.021873 12549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 330c48de0d714564b0a7c17e3f69028c. Latest consensus state: current_term: 1 leader_uuid: "330c48de0d714564b0a7c17e3f69028c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330c48de0d714564b0a7c17e3f69028c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } } }
I20260810 03:44:26.022082 12549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:26.022372 12551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "330c48de0d714564b0a7c17e3f69028c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330c48de0d714564b0a7c17e3f69028c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } } }
I20260810 03:44:26.022821 12551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:26.023337 12541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:26.017896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55999 (local address 127.0.0.1:42705)
0810 03:44:26.018007 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:26.018013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:26.018361 (+ 348us) server_negotiation.cc:407] Connection header received
0810 03:44:26.018492 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:26.018496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:26.018544 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:26.018613 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:26.018965 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:26.019556 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:26.020703 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:26.021028 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:26.021115 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:26.022823 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:26.022826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:26.022829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:26.022843 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:26.022876 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:26.022892 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:26.022982 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:26.023150 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 03:44:26.023209 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 03:44:26.022533 12555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:26.027148 12555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:26.033115 12555 catalog_manager.cc:1347] Generated new cluster ID: 468906065cd645f883407eba2e483921
I20260810 03:44:26.033196 12555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:26.053331 12555 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:26.054770 12555 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:26.066030 12555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 330c48de0d714564b0a7c17e3f69028c: Generated new TSK 0
I20260810 03:44:26.066881 12555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:26.076828 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 12459
2026-08-10T03:44:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1604 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:26 dist-test-slave-ss4n krb5kdc[12572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:26 dist-test-slave-ss4n krb5kdc[12572](info): set up 2 sockets
Aug 10 03:44:26 dist-test-slave-ss4n krb5kdc[12572](info): commencing operation
krb5kdc: starting...
W20260810 03:44:27.238015 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:27 dist-test-slave-ss4n krb5kdc[12572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:44:27.381966 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45719
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:27.498466 12589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:27.498811 12589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:27.498891 12589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:27.503018 12589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:27.503131 12589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:27.503172 12589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:27.503211 12589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:27.503245 12589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:27.508549 12589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:27.509708 12589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:27.510761 12589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:27.520493 12594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:27.520674 12595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:27.520606 12597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:27.520552 12589 server_base.cc:1034] running on GCE node
I20260810 03:44:27.521341 12589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:27.522629 12589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:27.523877 12589 hybrid_clock.cc:648] HybridClock initialized: now 1786333467523860 us; error 28 us; skew 500 ppm
Aug 10 03:44:27 dist-test-slave-ss4n krb5kdc[12572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:27.526914 12589 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:44:27.527801 12589 webserver.cc:460] Webserver started at http://127.0.0.1:43607/ using document root <none> and password file <none>
I20260810 03:44:27.528365 12589 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:27.528456 12589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:27.528702 12589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:27.530807 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "5888c1967f22456eb59d8adb89be6d48"
format_stamp: "Formatted at 2026-08-10 03:44:27 on dist-test-slave-ss4n"
server_key: "275fecde4911a0c50fe0226fd193c9be"
server_key_iv: "cfafee7d4b9d4bd18f44ff0a1f13ec0c"
server_key_version: "encryptionkey@0"
I20260810 03:44:27.531384 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "5888c1967f22456eb59d8adb89be6d48"
format_stamp: "Formatted at 2026-08-10 03:44:27 on dist-test-slave-ss4n"
server_key: "275fecde4911a0c50fe0226fd193c9be"
server_key_iv: "cfafee7d4b9d4bd18f44ff0a1f13ec0c"
server_key_version: "encryptionkey@0"
I20260810 03:44:27.535373 12589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:44:27.538173 12604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:27.539491 12589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:44:27.539667 12589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "5888c1967f22456eb59d8adb89be6d48"
format_stamp: "Formatted at 2026-08-10 03:44:27 on dist-test-slave-ss4n"
server_key: "275fecde4911a0c50fe0226fd193c9be"
server_key_iv: "cfafee7d4b9d4bd18f44ff0a1f13ec0c"
server_key_version: "encryptionkey@0"
I20260810 03:44:27.539793 12589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:27.544780 12589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:27.547974 12589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:27.548195 12589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:27.555542 12589 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45719
I20260810 03:44:27.555604 12665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45719 every 8 connection(s)
I20260810 03:44:27.556620 12589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:27.559540 12666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:27.559690 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 12589
I20260810 03:44:27.559842 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:27.560159 22205 external_mini_cluster.cc:1428] Setting key 0d75c6f4633b8aef25ca0845fbb9e394
I20260810 03:44:27.566431 12669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:27.561860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59045 (local address 127.0.0.1:45719)
0810 03:44:27.562369 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:27.562389 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:27.562435 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:44:27.563221 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:27.563247 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:27.563634 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:27.564026 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:27.564901 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:27.564907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:27.564922 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:27.564935 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:27.565004 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:27.565032 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:27.565193 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:27.565272 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 03:44:27.565432 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":137,"threads_started":1}
I20260810 03:44:27.567409 12666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48: Bootstrap starting.
I20260810 03:44:27.569630 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:44:27.570013 12666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:27.570751 12666 log.cc:824] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:27.573102 12666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48: No bootstrap required, opened a new log
I20260810 03:44:27.573529 12669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:27.571773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45065 (local address 127.0.0.1:45719)
0810 03:44:27.571907 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:27.571911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:27.572126 (+ 215us) server_negotiation.cc:407] Connection header received
0810 03:44:27.572255 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:27.572257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:27.572286 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:27.572352 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:27.573077 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:27.573080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:27.573082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:27.573092 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:27.573121 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:27.573133 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:27.573243 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:27.573341 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 03:44:27.573382 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:44:27.577075 12666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5888c1967f22456eb59d8adb89be6d48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45719 } }
I20260810 03:44:27.577396 12666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:27.577534 12666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5888c1967f22456eb59d8adb89be6d48, State: Initialized, Role: FOLLOWER
I20260810 03:44:27.578056 12666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5888c1967f22456eb59d8adb89be6d48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45719 } }
I20260810 03:44:27.578189 12666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:27.578276 12666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:27.578378 12666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:27.579340 12666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5888c1967f22456eb59d8adb89be6d48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45719 } }
I20260810 03:44:27.579698 12666 leader_election.cc:302] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5888c1967f22456eb59d8adb89be6d48; no voters:
I20260810 03:44:27.580029 12666 leader_election.cc:288] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:27.580286 12676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:27.580597 12676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [term 1 LEADER]: Becoming Leader. State: Replica: 5888c1967f22456eb59d8adb89be6d48, State: Running, Role: LEADER
I20260810 03:44:27.580966 12676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5888c1967f22456eb59d8adb89be6d48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45719 } }
I20260810 03:44:27.581394 12666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:27.582839 12678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5888c1967f22456eb59d8adb89be6d48. Latest consensus state: current_term: 1 leader_uuid: "5888c1967f22456eb59d8adb89be6d48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5888c1967f22456eb59d8adb89be6d48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45719 } } }
I20260810 03:44:27.582969 12678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:27.582911 12677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5888c1967f22456eb59d8adb89be6d48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5888c1967f22456eb59d8adb89be6d48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45719 } } }
I20260810 03:44:27.583040 12677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:27.583446 12687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:27.587250 12687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:27.593565 12687 catalog_manager.cc:1347] Generated new cluster ID: fe8c023cf34141b4ad99b67c0a323181
I20260810 03:44:27.593672 12687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:27.608924 12687 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:27.609400 12687 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:27.613744 12687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5888c1967f22456eb59d8adb89be6d48: Generated new TSK 0
I20260810 03:44:27.614465 12687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:27.622903 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 12589
2026-08-10T03:44:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1547 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:27 dist-test-slave-ss4n krb5kdc[12701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:27 dist-test-slave-ss4n krb5kdc[12701](info): set up 2 sockets
Aug 10 03:44:27 dist-test-slave-ss4n krb5kdc[12701](info): commencing operation
krb5kdc: starting...
W20260810 03:44:28.797796 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:28 dist-test-slave-ss4n krb5kdc[12701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:44:28.934237 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41157
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:29.049165 12714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:29.049461 12714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:29.049523 12714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:29.053712 12714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:29.053793 12714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:29.053813 12714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:29.053833 12714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:29.053848 12714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:29.059473 12714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:29.060701 12714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:29.061829 12714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:29.072063 12720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:29.072062 12719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:29.072259 12722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:29.072268 12714 server_base.cc:1034] running on GCE node
I20260810 03:44:29.073098 12714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:29.074414 12714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:29.075644 12714 hybrid_clock.cc:648] HybridClock initialized: now 1786333469075627 us; error 39 us; skew 500 ppm
Aug 10 03:44:29 dist-test-slave-ss4n krb5kdc[12701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:29.078838 12714 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:44:29.079840 12714 webserver.cc:460] Webserver started at http://127.0.0.1:36141/ using document root <none> and password file <none>
I20260810 03:44:29.080420 12714 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:29.080515 12714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:29.080766 12714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:29.082945 12714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "462bbd54e60e48c7b18627ac8cb0d154"
format_stamp: "Formatted at 2026-08-10 03:44:29 on dist-test-slave-ss4n"
server_key: "32aae0c7e97c17c8af99acb6269b2468"
server_key_iv: "3d52240ca8725a4a098552eb8e1100cb"
server_key_version: "encryptionkey@0"
I20260810 03:44:29.083526 12714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "462bbd54e60e48c7b18627ac8cb0d154"
format_stamp: "Formatted at 2026-08-10 03:44:29 on dist-test-slave-ss4n"
server_key: "32aae0c7e97c17c8af99acb6269b2468"
server_key_iv: "3d52240ca8725a4a098552eb8e1100cb"
server_key_version: "encryptionkey@0"
I20260810 03:44:29.087311 12714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:44:29.089888 12729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:29.091229 12714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:44:29.091403 12714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "462bbd54e60e48c7b18627ac8cb0d154"
format_stamp: "Formatted at 2026-08-10 03:44:29 on dist-test-slave-ss4n"
server_key: "32aae0c7e97c17c8af99acb6269b2468"
server_key_iv: "3d52240ca8725a4a098552eb8e1100cb"
server_key_version: "encryptionkey@0"
I20260810 03:44:29.091529 12714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:29.107256 12714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:29.110422 12714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:29.110658 12714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:29.117798 12714 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41157
I20260810 03:44:29.117835 12788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41157 every 8 connection(s)
I20260810 03:44:29.118793 12714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:29.121520 12789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:29.122025 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 12714
I20260810 03:44:29.122232 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:29.122550 22205 external_mini_cluster.cc:1428] Setting key 1880caedc3563de285b3869c0cb10e42
I20260810 03:44:29.127591 12789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154: Bootstrap starting.
I20260810 03:44:29.130016 12789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:29.130820 12789 log.cc:824] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:44:29 dist-test-slave-ss4n krb5kdc[12701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 03:44:29.132865 12789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154: No bootstrap required, opened a new log
I20260810 03:44:29.136426 12789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462bbd54e60e48c7b18627ac8cb0d154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41157 } }
I20260810 03:44:29.136715 12789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:29.136807 12789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 462bbd54e60e48c7b18627ac8cb0d154, State: Initialized, Role: FOLLOWER
I20260810 03:44:29.137255 12789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462bbd54e60e48c7b18627ac8cb0d154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41157 } }
I20260810 03:44:29.137374 12789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:29.137460 12789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:29.137609 12789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:29.137619 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:29.124032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:36083 (local address 127.0.0.1:41157)
0810 03:44:29.124501 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:29.124516 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:29.124561 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:44:29.125270 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:29.125295 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:29.125770 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:29.126128 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:29.126981 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:29.128035 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:29.128948 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:29.129429 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:29.129646 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:29.132454 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:29.132460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:29.132475 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:29.132503 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:29.134352 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:29.134880 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:29.134885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:29.134890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:29.135000 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:29.135353 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:29.135361 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:29.135363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:29.135760 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:29.135999 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:29.136343 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 03:44:29.136577 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":136,"threads_started":1}
I20260810 03:44:29.138576 12789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462bbd54e60e48c7b18627ac8cb0d154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41157 } }
I20260810 03:44:29.138936 12789 leader_election.cc:302] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 462bbd54e60e48c7b18627ac8cb0d154; no voters:
I20260810 03:44:29.139292 12789 leader_election.cc:288] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:29.139405 12794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:29.139699 12794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [term 1 LEADER]: Becoming Leader. State: Replica: 462bbd54e60e48c7b18627ac8cb0d154, State: Running, Role: LEADER
I20260810 03:44:29.140066 12794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462bbd54e60e48c7b18627ac8cb0d154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41157 } }
I20260810 03:44:29.140637 12789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:29.140980 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:44:29.141981 12796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 462bbd54e60e48c7b18627ac8cb0d154. Latest consensus state: current_term: 1 leader_uuid: "462bbd54e60e48c7b18627ac8cb0d154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462bbd54e60e48c7b18627ac8cb0d154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41157 } } }
I20260810 03:44:29.141978 12795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "462bbd54e60e48c7b18627ac8cb0d154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462bbd54e60e48c7b18627ac8cb0d154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41157 } } }
I20260810 03:44:29.142282 12795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:29.142282 12796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:29.142774 12809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:29.145875 12809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:29.152165 12809 catalog_manager.cc:1347] Generated new cluster ID: 03bd291859f64959a408545518d6144b
I20260810 03:44:29.152279 12809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:29.154042 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:29.143426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:37975 (local address 127.0.0.1:41157)
0810 03:44:29.143545 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:29.143549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:29.143839 (+ 290us) server_negotiation.cc:407] Connection header received
0810 03:44:29.144023 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:29.144028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:29.144079 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:29.144136 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:29.145138 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:29.145965 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:29.147135 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:29.147489 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:29.147569 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:29.149564 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:29.149568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:29.149571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:29.149600 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:29.151849 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:29.152475 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:29.152479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:29.152481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:29.152582 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:29.152960 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:29.152963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:29.152965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:29.153184 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:29.153373 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:29.153739 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 03:44:29.153849 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 03:44:29.163931 12809 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:29.165030 12809 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:29.171089 12809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 462bbd54e60e48c7b18627ac8cb0d154: Generated new TSK 0
I20260810 03:44:29.171877 12809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:29.176421 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 12714
W20260810 03:44:29.185410 12696 connection.cc:535] client connection to 127.0.0.1:41157 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41157): Connection reset by peer (error 104)
2026-08-10T03:44:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1553 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:29 dist-test-slave-ss4n krb5kdc[12823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:29 dist-test-slave-ss4n krb5kdc[12823](info): set up 2 sockets
Aug 10 03:44:29 dist-test-slave-ss4n krb5kdc[12823](info): commencing operation
krb5kdc: starting...
W20260810 03:44:30.318074 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:30 dist-test-slave-ss4n krb5kdc[12823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:44:30.461292 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:30.580718 12838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:30.581034 12838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:30.581079 12838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:30.585273 12838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:30.585356 12838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:30.585376 12838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:30.585393 12838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:30.585409 12838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:30.590970 12838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:30.592163 12838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:30.593294 12838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:30.602365 12844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:30.602380 12845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:30.602389 12847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:30.602864 12838 server_base.cc:1034] running on GCE node
I20260810 03:44:30.603506 12838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:30.604681 12838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:30.605840 12838 hybrid_clock.cc:648] HybridClock initialized: now 1786333470605812 us; error 46 us; skew 500 ppm
Aug 10 03:44:30 dist-test-slave-ss4n krb5kdc[12823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:30.608744 12838 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:44:30.609798 12838 webserver.cc:460] Webserver started at http://127.0.0.1:38461/ using document root <none> and password file <none>
I20260810 03:44:30.610374 12838 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:30.610466 12838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:30.610703 12838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:30.612813 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "f98ce06e9ebb4957ba0445894fe9797e"
format_stamp: "Formatted at 2026-08-10 03:44:30 on dist-test-slave-ss4n"
server_key: "91f32deb526ee6e64ad1aa3c704db0fd"
server_key_iv: "9381542d4425f1c1a27959029def48bb"
server_key_version: "encryptionkey@0"
I20260810 03:44:30.613395 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "f98ce06e9ebb4957ba0445894fe9797e"
format_stamp: "Formatted at 2026-08-10 03:44:30 on dist-test-slave-ss4n"
server_key: "91f32deb526ee6e64ad1aa3c704db0fd"
server_key_iv: "9381542d4425f1c1a27959029def48bb"
server_key_version: "encryptionkey@0"
I20260810 03:44:30.617148 12838 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:44:30.619731 12855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:30.621081 12838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:44:30.621260 12838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "f98ce06e9ebb4957ba0445894fe9797e"
format_stamp: "Formatted at 2026-08-10 03:44:30 on dist-test-slave-ss4n"
server_key: "91f32deb526ee6e64ad1aa3c704db0fd"
server_key_iv: "9381542d4425f1c1a27959029def48bb"
server_key_version: "encryptionkey@0"
I20260810 03:44:30.621418 12838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:30.652585 12838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:30.656523 12838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:30.656802 12838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:30.664654 12838 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44769
I20260810 03:44:30.664718 12916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44769 every 8 connection(s)
I20260810 03:44:30.665783 12838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:30.668927 12917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:30.670130 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 12838
I20260810 03:44:30.670289 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:30.670629 22205 external_mini_cluster.cc:1428] Setting key bbd907c17844cccc60fb80165a679ad7
I20260810 03:44:30.676170 12917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e: Bootstrap starting.
I20260810 03:44:30.678664 12917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:30.679435 12917 log.cc:824] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:30.680753 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:30.672168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:45757 (local address 127.0.0.1:44769)
0810 03:44:30.672674 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:30.672691 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:30.672727 (+ 36us) server_negotiation.cc:407] Connection header received
0810 03:44:30.673585 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:30.673617 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:30.674210 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:30.674669 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:30.675110 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:30.676081 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:30.677078 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:30.677645 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:30.677817 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:30.678840 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:30.678845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:30.678858 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:30.679290 (+ 432us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:30.679353 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:30.679376 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:30.679521 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:30.679731 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 03:44:30.679881 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":149,"threads_started":1}
I20260810 03:44:30.681767 12917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e: No bootstrap required, opened a new log
I20260810 03:44:30.683802 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:44:30.685830 12917 raft_consensus.cc:348] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98ce06e9ebb4957ba0445894fe9797e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44769 } }
I20260810 03:44:30.686157 12917 raft_consensus.cc:374] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:30.686237 12917 raft_consensus.cc:729] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f98ce06e9ebb4957ba0445894fe9797e, State: Initialized, Role: FOLLOWER
I20260810 03:44:30.686748 12917 consensus_queue.cc:260] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98ce06e9ebb4957ba0445894fe9797e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44769 } }
I20260810 03:44:30.686944 12917 raft_consensus.cc:388] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:30.687039 12917 raft_consensus.cc:482] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:30.687142 12917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:30.688526 12917 raft_consensus.cc:504] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98ce06e9ebb4957ba0445894fe9797e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44769 } }
I20260810 03:44:30.688913 12917 leader_election.cc:302] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f98ce06e9ebb4957ba0445894fe9797e; no voters:
I20260810 03:44:30.689285 12917 leader_election.cc:288] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:30.689383 12929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:30.689677 12929 raft_consensus.cc:686] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [term 1 LEADER]: Becoming Leader. State: Replica: f98ce06e9ebb4957ba0445894fe9797e, State: Running, Role: LEADER
I20260810 03:44:30.690023 12929 consensus_queue.cc:237] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98ce06e9ebb4957ba0445894fe9797e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44769 } }
I20260810 03:44:30.690791 12917 sys_catalog.cc:564] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:30.690902 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:30.685873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:33833 (local address 127.0.0.1:44769)
0810 03:44:30.686013 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:30.686017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:30.686255 (+ 238us) server_negotiation.cc:407] Connection header received
0810 03:44:30.686377 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:30.686385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:30.686426 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:30.686526 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:30.686827 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:30.687524 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:30.688814 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:30.689048 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:30.689169 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:30.690209 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:30.690213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:30.690216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:30.690230 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:30.690271 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:30.690284 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:30.690349 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:30.690662 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 03:44:30.690705 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 03:44:30.692389 12932 sys_catalog.cc:455] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f98ce06e9ebb4957ba0445894fe9797e. Latest consensus state: current_term: 1 leader_uuid: "f98ce06e9ebb4957ba0445894fe9797e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98ce06e9ebb4957ba0445894fe9797e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44769 } } }
I20260810 03:44:30.692560 12932 sys_catalog.cc:458] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:30.692965 12945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:30.692922 12931 sys_catalog.cc:455] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f98ce06e9ebb4957ba0445894fe9797e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98ce06e9ebb4957ba0445894fe9797e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44769 } } }
I20260810 03:44:30.693008 12931 sys_catalog.cc:458] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:30.695753 12945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:30.701448 12945 catalog_manager.cc:1347] Generated new cluster ID: c2d4c9845cd74fd5bb07ab42bff809d9
I20260810 03:44:30.701597 12945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:30.722446 12945 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:30.723459 12945 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:30.730213 12945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f98ce06e9ebb4957ba0445894fe9797e: Generated new TSK 0
I20260810 03:44:30.730947 12945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:30.740792 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 12838
W20260810 03:44:30.749658 12821 connection.cc:535] client connection to 127.0.0.1:44769 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44769): Connection reset by peer (error 104)
2026-08-10T03:44:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1564 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:30 dist-test-slave-ss4n krb5kdc[12953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:30 dist-test-slave-ss4n krb5kdc[12953](info): set up 2 sockets
Aug 10 03:44:30 dist-test-slave-ss4n krb5kdc[12953](info): commencing operation
krb5kdc: starting...
W20260810 03:44:31.858940 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:31 dist-test-slave-ss4n krb5kdc[12953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:44:32.009245 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:32.136960 12967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:32.137245 12967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:32.137290 12967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:32.141372 12967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:32.141446 12967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:32.141467 12967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:32.141544 12967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:32.141582 12967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:32.146968 12967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:32.148079 12967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:32.149233 12967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:32.158280 12972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:32.158262 12973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:32.158381 12967 server_base.cc:1034] running on GCE node
W20260810 03:44:32.158284 12975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:32.159132 12967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:32.160283 12967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:32.161556 12967 hybrid_clock.cc:648] HybridClock initialized: now 1786333472161534 us; error 35 us; skew 500 ppm
Aug 10 03:44:32 dist-test-slave-ss4n krb5kdc[12953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:32.168329 12967 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:44:32.169391 12967 webserver.cc:460] Webserver started at http://127.0.0.1:38965/ using document root <none> and password file <none>
I20260810 03:44:32.170078 12967 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:32.170146 12967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:32.170325 12967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:32.172422 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "229be537af0a48b496af114e35b939a7"
format_stamp: "Formatted at 2026-08-10 03:44:32 on dist-test-slave-ss4n"
server_key: "e95ab7d10c5ae1be9547d22aa76e80e9"
server_key_iv: "cf3a14b24d228fb3ec5faac120f381d6"
server_key_version: "encryptionkey@0"
I20260810 03:44:32.172950 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "229be537af0a48b496af114e35b939a7"
format_stamp: "Formatted at 2026-08-10 03:44:32 on dist-test-slave-ss4n"
server_key: "e95ab7d10c5ae1be9547d22aa76e80e9"
server_key_iv: "cf3a14b24d228fb3ec5faac120f381d6"
server_key_version: "encryptionkey@0"
I20260810 03:44:32.176987 12967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:44:32.179661 12982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:32.180985 12967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:44:32.181089 12967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "229be537af0a48b496af114e35b939a7"
format_stamp: "Formatted at 2026-08-10 03:44:32 on dist-test-slave-ss4n"
server_key: "e95ab7d10c5ae1be9547d22aa76e80e9"
server_key_iv: "cf3a14b24d228fb3ec5faac120f381d6"
server_key_version: "encryptionkey@0"
I20260810 03:44:32.181237 12967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:32.186021 12967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:32.189385 12967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:32.189615 12967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:32.196805 12967 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33723
I20260810 03:44:32.196878 13040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33723 every 8 connection(s)
I20260810 03:44:32.197840 12967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:32.200739 13041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:32.206475 13041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7: Bootstrap starting.
I20260810 03:44:32.207336 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 12967
I20260810 03:44:32.207484 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:32.207837 22205 external_mini_cluster.cc:1428] Setting key c3709dfb2670cb94bf6df8008d44aac3
I20260810 03:44:32.209653 13041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:32.210424 13041 log.cc:824] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:32.212736 13041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7: No bootstrap required, opened a new log
I20260810 03:44:32.213557 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:32.209052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:58659 (local address 127.0.0.1:33723)
0810 03:44:32.209621 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:32.209645 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:32.209691 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:44:32.210602 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:32.210625 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:32.210988 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:32.211352 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:32.212155 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:32.212172 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:32.212190 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:32.212207 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:32.212283 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:32.212306 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:32.212432 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:32.212543 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 03:44:32.212710 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":130,"threads_started":1}
I20260810 03:44:32.216574 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:44:32.217036 13041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229be537af0a48b496af114e35b939a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } }
I20260810 03:44:32.217294 13041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:32.217386 13041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229be537af0a48b496af114e35b939a7, State: Initialized, Role: FOLLOWER
I20260810 03:44:32.217898 13041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229be537af0a48b496af114e35b939a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } }
I20260810 03:44:32.218067 13041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:32.218166 13041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:32.218269 13041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:32.219620 13041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229be537af0a48b496af114e35b939a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } }
I20260810 03:44:32.220027 13041 leader_election.cc:302] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 229be537af0a48b496af114e35b939a7; no voters:
I20260810 03:44:32.220432 13041 leader_election.cc:288] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:32.220487 13053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:32.220754 13053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [term 1 LEADER]: Becoming Leader. State: Replica: 229be537af0a48b496af114e35b939a7, State: Running, Role: LEADER
I20260810 03:44:32.220919 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:32.218852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:33851 (local address 127.0.0.1:33723)
0810 03:44:32.218981 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:32.218984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:32.219202 (+ 218us) server_negotiation.cc:407] Connection header received
0810 03:44:32.219333 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:32.219335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:32.219364 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:32.219432 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:32.220357 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:32.220360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:32.220362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:32.220372 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:32.220396 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:32.220406 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:32.220555 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:32.220706 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 03:44:32.220753 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 03:44:32.221120 13053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229be537af0a48b496af114e35b939a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } }
I20260810 03:44:32.221819 13041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:32.222430 13056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "229be537af0a48b496af114e35b939a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229be537af0a48b496af114e35b939a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } } }
I20260810 03:44:32.222549 13056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:32.222774 13056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 229be537af0a48b496af114e35b939a7. Latest consensus state: current_term: 1 leader_uuid: "229be537af0a48b496af114e35b939a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229be537af0a48b496af114e35b939a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } } }
I20260810 03:44:32.222860 13056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:32.223258 13059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:32.226645 13059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:32.232333 13059 catalog_manager.cc:1347] Generated new cluster ID: bfa9f5f3d77245ecad2485af9b38e002
I20260810 03:44:32.232515 13059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:32.258970 13059 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:32.259369 13059 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:32.264943 13059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 229be537af0a48b496af114e35b939a7: Generated new TSK 0
I20260810 03:44:32.265877 13059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:32.278661 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 12967
2026-08-10T03:44:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1536 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:32 dist-test-slave-ss4n krb5kdc[13075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:32 dist-test-slave-ss4n krb5kdc[13075](info): set up 2 sockets
Aug 10 03:44:32 dist-test-slave-ss4n krb5kdc[13075](info): commencing operation
krb5kdc: starting...
W20260810 03:44:33.424657 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:33 dist-test-slave-ss4n krb5kdc[13075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:44:33.561621 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38447
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:33.678802 13089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:33.679085 13089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:33.679126 13089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:33.683292 13089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:33.683364 13089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:33.683384 13089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:33.683401 13089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:33.683416 13089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:33.688851 13089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38447
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:33.690191 13089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:33.691270 13089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:33.700426 13098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:33.700423 13095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:33.700498 13096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:33.700879 13089 server_base.cc:1034] running on GCE node
I20260810 03:44:33.701638 13089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:33.702890 13089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:33.704142 13089 hybrid_clock.cc:648] HybridClock initialized: now 1786333473704122 us; error 36 us; skew 500 ppm
Aug 10 03:44:33 dist-test-slave-ss4n krb5kdc[13075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:33.707372 13089 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:44:33.708424 13089 webserver.cc:460] Webserver started at http://127.0.0.1:46817/ using document root <none> and password file <none>
I20260810 03:44:33.709100 13089 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:33.709172 13089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:33.709404 13089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:33.711683 13089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "9d3fa5f3ffc3463f81aee19a142afbca"
format_stamp: "Formatted at 2026-08-10 03:44:33 on dist-test-slave-ss4n"
server_key: "71ef72aba0630d4ffb1c3b8b56435921"
server_key_iv: "6c2a8d1cf401753798e22af349a5bd4b"
server_key_version: "encryptionkey@0"
I20260810 03:44:33.712347 13089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "9d3fa5f3ffc3463f81aee19a142afbca"
format_stamp: "Formatted at 2026-08-10 03:44:33 on dist-test-slave-ss4n"
server_key: "71ef72aba0630d4ffb1c3b8b56435921"
server_key_iv: "6c2a8d1cf401753798e22af349a5bd4b"
server_key_version: "encryptionkey@0"
I20260810 03:44:33.716509 13089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260810 03:44:33.719267 13105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:33.720548 13089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:44:33.720685 13089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "9d3fa5f3ffc3463f81aee19a142afbca"
format_stamp: "Formatted at 2026-08-10 03:44:33 on dist-test-slave-ss4n"
server_key: "71ef72aba0630d4ffb1c3b8b56435921"
server_key_iv: "6c2a8d1cf401753798e22af349a5bd4b"
server_key_version: "encryptionkey@0"
I20260810 03:44:33.720804 13089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:33.741694 13089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:33.745098 13089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:33.745338 13089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:33.753171 13089 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38447
I20260810 03:44:33.753196 13160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38447 every 8 connection(s)
I20260810 03:44:33.754338 13089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:33.757266 13161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:33.759876 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 13089
I20260810 03:44:33.760072 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:33.760407 22205 external_mini_cluster.cc:1428] Setting key 5bc558818a492765d13611a17c69730b
I20260810 03:44:33.763475 13161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca: Bootstrap starting.
I20260810 03:44:33.765805 13161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:33.766552 13161 log.cc:824] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:33.768697 13161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca: No bootstrap required, opened a new log
Aug 10 03:44:33 dist-test-slave-ss4n krb5kdc[13075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 03:44:33.773303 13161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d3fa5f3ffc3463f81aee19a142afbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38447 } }
I20260810 03:44:33.773676 13161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:33.773787 13161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d3fa5f3ffc3463f81aee19a142afbca, State: Initialized, Role: FOLLOWER
I20260810 03:44:33.774333 13161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d3fa5f3ffc3463f81aee19a142afbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38447 } }
I20260810 03:44:33.774513 13161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:33.774595 13161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:33.774755 13161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:33.776173 13161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d3fa5f3ffc3463f81aee19a142afbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38447 } }
I20260810 03:44:33.776597 13161 leader_election.cc:302] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d3fa5f3ffc3463f81aee19a142afbca; no voters:
I20260810 03:44:33.777027 13161 leader_election.cc:288] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:33.777180 13166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:33.777424 13166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [term 1 LEADER]: Becoming Leader. State: Replica: 9d3fa5f3ffc3463f81aee19a142afbca, State: Running, Role: LEADER
I20260810 03:44:33.777455 13164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:33.761962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:38639 (local address 127.0.0.1:38447)
0810 03:44:33.762467 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:33.762483 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:33.762521 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:44:33.763206 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:33.763225 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:33.763632 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:33.764027 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:33.764945 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:33.766209 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:33.767445 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:33.767968 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:33.768166 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:33.771137 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:33.771142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:33.771157 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:33.771188 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:33.773314 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:33.774166 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:33.774172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:33.774177 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:33.774300 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:33.774720 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:33.774724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:33.774726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:33.775353 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:33.775612 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:33.775989 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 03:44:33.776270 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":164,"threads_started":1}
I20260810 03:44:33.777837 13166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d3fa5f3ffc3463f81aee19a142afbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38447 } }
I20260810 03:44:33.778527 13161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:33.779696 13168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d3fa5f3ffc3463f81aee19a142afbca. Latest consensus state: current_term: 1 leader_uuid: "9d3fa5f3ffc3463f81aee19a142afbca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d3fa5f3ffc3463f81aee19a142afbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38447 } } }
I20260810 03:44:33.779834 13168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:33.780205 13167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d3fa5f3ffc3463f81aee19a142afbca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d3fa5f3ffc3463f81aee19a142afbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38447 } } }
I20260810 03:44:33.780289 13167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:33.781559 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 03:44:33.784978 13186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:44:33.785070 13186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:44:33.785140 13173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:33.786128 13173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:33.793853 13173 catalog_manager.cc:1347] Generated new cluster ID: ce1f00631c264f29b75dc24c67d70332
I20260810 03:44:33.793941 13173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:33.794852 13164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:33.783795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:42003 (local address 127.0.0.1:38447)
0810 03:44:33.783932 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:33.783937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:33.785794 (+ 1857us) server_negotiation.cc:407] Connection header received
0810 03:44:33.785963 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:33.785966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:33.786023 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:33.786089 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:33.786879 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:33.787446 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:33.788440 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:33.788845 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:33.788984 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:33.791065 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:33.791069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:33.791072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:33.791100 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:33.792755 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:33.793293 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:33.793298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:33.793300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:33.793390 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:33.793786 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:33.793790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:33.793792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:33.794061 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:33.794240 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:33.794566 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 03:44:33.794670 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 03:44:33.801426 13173 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:33.802565 13173 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:33.820250 13173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d3fa5f3ffc3463f81aee19a142afbca: Generated new TSK 0
I20260810 03:44:33.821067 13173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:33.828979 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 13089
W20260810 03:44:33.838778 13073 connection.cc:535] client connection to 127.0.0.1:38447 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38447): Connection reset by peer (error 104)
2026-08-10T03:44:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1552 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:33 dist-test-slave-ss4n krb5kdc[13197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:33 dist-test-slave-ss4n krb5kdc[13197](info): set up 2 sockets
Aug 10 03:44:33 dist-test-slave-ss4n krb5kdc[13197](info): commencing operation
krb5kdc: starting...
W20260810 03:44:34.965512 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:35 dist-test-slave-ss4n krb5kdc[13197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:44:35.105273 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:35.220206 13212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:35.220481 13212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:35.220522 13212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:35.224471 13212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:35.224530 13212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:35.224550 13212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:35.224566 13212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:35.224582 13212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:35.229745 13212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:35.230747 13212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:35.231889 13212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:35.241181 13218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:35.241245 13219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:35.241221 13212 server_base.cc:1034] running on GCE node
W20260810 03:44:35.241178 13221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:35.241995 13212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:35.243155 13212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:35.244341 13212 hybrid_clock.cc:648] HybridClock initialized: now 1786333475244328 us; error 29 us; skew 500 ppm
Aug 10 03:44:35 dist-test-slave-ss4n krb5kdc[13197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:35.247152 13212 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:44:35.247984 13212 webserver.cc:460] Webserver started at http://127.0.0.1:45897/ using document root <none> and password file <none>
I20260810 03:44:35.248517 13212 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:35.248576 13212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:35.248741 13212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:35.250859 13212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "c0487aad8b494c7abd99f019783cca5d"
format_stamp: "Formatted at 2026-08-10 03:44:35 on dist-test-slave-ss4n"
server_key: "8cf246c80b81804b04f6423b5e591203"
server_key_iv: "02172cef5ac4b434969117981440dc77"
server_key_version: "encryptionkey@0"
I20260810 03:44:35.251394 13212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "c0487aad8b494c7abd99f019783cca5d"
format_stamp: "Formatted at 2026-08-10 03:44:35 on dist-test-slave-ss4n"
server_key: "8cf246c80b81804b04f6423b5e591203"
server_key_iv: "02172cef5ac4b434969117981440dc77"
server_key_version: "encryptionkey@0"
I20260810 03:44:35.255095 13212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:44:35.257401 13229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:35.258713 13212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 03:44:35.258812 13212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "c0487aad8b494c7abd99f019783cca5d"
format_stamp: "Formatted at 2026-08-10 03:44:35 on dist-test-slave-ss4n"
server_key: "8cf246c80b81804b04f6423b5e591203"
server_key_iv: "02172cef5ac4b434969117981440dc77"
server_key_version: "encryptionkey@0"
I20260810 03:44:35.258894 13212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:35.278059 13212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:35.281113 13212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:35.281270 13212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:35.288830 13212 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36053
I20260810 03:44:35.288868 13287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36053 every 8 connection(s)
I20260810 03:44:35.289893 13212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:35.292662 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 13212
I20260810 03:44:35.292804 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:35.292856 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:35.293088 22205 external_mini_cluster.cc:1428] Setting key a6d86ce221abaa612edc681174733829
I20260810 03:44:35.300452 13288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d: Bootstrap starting.
I20260810 03:44:35.303409 13288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:35.303623 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:35.294550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:52551 (local address 127.0.0.1:36053)
0810 03:44:35.295000 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:35.295019 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:35.295064 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:44:35.295763 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:35.295788 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:35.296254 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:35.296642 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:35.297086 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:35.298598 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:35.299490 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:35.300175 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:35.300342 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:35.301476 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:35.301516 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:35.301532 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:35.302074 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:35.302142 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:35.302191 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:35.302326 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:35.302538 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 03:44:35.302673 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":122,"threads_started":1}
I20260810 03:44:35.304265 13288 log.cc:824] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:35.306408 13288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d: No bootstrap required, opened a new log
I20260810 03:44:35.306702 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:44:35.311275 13288 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0487aad8b494c7abd99f019783cca5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36053 } }
I20260810 03:44:35.311561 13288 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:35.311618 13288 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0487aad8b494c7abd99f019783cca5d, State: Initialized, Role: FOLLOWER
I20260810 03:44:35.312130 13288 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0487aad8b494c7abd99f019783cca5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36053 } }
I20260810 03:44:35.312264 13288 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:35.312351 13288 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:35.312475 13288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:35.313333 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:35.309036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:36267 (local address 127.0.0.1:36053)
0810 03:44:35.309175 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:35.309179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:35.309409 (+ 230us) server_negotiation.cc:407] Connection header received
0810 03:44:35.309537 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:35.309541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:35.309594 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:35.309681 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:35.309940 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:35.310723 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:35.311541 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:35.311877 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:35.311985 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:35.312880 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:35.312884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:35.312887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:35.312900 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:35.312927 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:35.312940 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:35.313038 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:35.313173 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 03:44:35.313209 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:44:35.313827 13288 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0487aad8b494c7abd99f019783cca5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36053 } }
I20260810 03:44:35.314217 13288 leader_election.cc:302] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0487aad8b494c7abd99f019783cca5d; no voters:
I20260810 03:44:35.314603 13288 leader_election.cc:288] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:35.314718 13300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:35.315001 13300 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [term 1 LEADER]: Becoming Leader. State: Replica: c0487aad8b494c7abd99f019783cca5d, State: Running, Role: LEADER
I20260810 03:44:35.315363 13300 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0487aad8b494c7abd99f019783cca5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36053 } }
I20260810 03:44:35.315927 13288 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:35.317058 13302 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0487aad8b494c7abd99f019783cca5d. Latest consensus state: current_term: 1 leader_uuid: "c0487aad8b494c7abd99f019783cca5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0487aad8b494c7abd99f019783cca5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36053 } } }
I20260810 03:44:35.317198 13302 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:35.317615 13310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:35.317582 13301 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0487aad8b494c7abd99f019783cca5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0487aad8b494c7abd99f019783cca5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36053 } } }
I20260810 03:44:35.317726 13301 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:35.321151 13310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:35.327618 13310 catalog_manager.cc:1347] Generated new cluster ID: 31860ea2e8434048a01ac3097d1e2c1b
I20260810 03:44:35.327698 13310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:35.336936 13310 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:35.338308 13310 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:35.351310 13310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0487aad8b494c7abd99f019783cca5d: Generated new TSK 0
I20260810 03:44:35.352320 13310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:35.360999 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 13212
W20260810 03:44:35.368860 13195 connection.cc:535] client connection to 127.0.0.1:36053 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36053): Connection reset by peer (error 104)
2026-08-10T03:44:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1530 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:35 dist-test-slave-ss4n krb5kdc[13325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:35 dist-test-slave-ss4n krb5kdc[13325](info): set up 2 sockets
Aug 10 03:44:35 dist-test-slave-ss4n krb5kdc[13325](info): commencing operation
krb5kdc: starting...
W20260810 03:44:36.548568 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:36 dist-test-slave-ss4n krb5kdc[13325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:44:36.689607 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:36.808210 13340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:36.808502 13340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:36.808547 13340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:36.812521 13340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:36.812584 13340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:36.812603 13340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:36.812621 13340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:36.812636 13340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:36.817874 13340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43653
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:36.818872 13340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:36.819825 13340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:36.828791 13349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:36.828827 13345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:36.828953 13340 server_base.cc:1034] running on GCE node
W20260810 03:44:36.828794 13346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:36.829756 13340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:36.830891 13340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:36.832093 13340 hybrid_clock.cc:648] HybridClock initialized: now 1786333476832076 us; error 34 us; skew 500 ppm
Aug 10 03:44:36 dist-test-slave-ss4n krb5kdc[13325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:36.835059 13340 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:44:36.835935 13340 webserver.cc:460] Webserver started at http://127.0.0.1:42361/ using document root <none> and password file <none>
I20260810 03:44:36.836453 13340 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:36.836513 13340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:36.836790 13340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:36.838943 13340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "c41351887fcc43a3b9a24e9fa6a4c533"
format_stamp: "Formatted at 2026-08-10 03:44:36 on dist-test-slave-ss4n"
server_key: "d220325a9a2cecda00882f820e32b7e0"
server_key_iv: "8ab8897f9f44c3a9b9f4ecbd01625bdb"
server_key_version: "encryptionkey@0"
I20260810 03:44:36.839540 13340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "c41351887fcc43a3b9a24e9fa6a4c533"
format_stamp: "Formatted at 2026-08-10 03:44:36 on dist-test-slave-ss4n"
server_key: "d220325a9a2cecda00882f820e32b7e0"
server_key_iv: "8ab8897f9f44c3a9b9f4ecbd01625bdb"
server_key_version: "encryptionkey@0"
I20260810 03:44:36.843463 13340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:44:36.846120 13356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:36.847486 13340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:44:36.847667 13340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "c41351887fcc43a3b9a24e9fa6a4c533"
format_stamp: "Formatted at 2026-08-10 03:44:36 on dist-test-slave-ss4n"
server_key: "d220325a9a2cecda00882f820e32b7e0"
server_key_iv: "8ab8897f9f44c3a9b9f4ecbd01625bdb"
server_key_version: "encryptionkey@0"
I20260810 03:44:36.847788 13340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:36.852401 13340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:36.855760 13340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:36.855955 13340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:36.863530 13340 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43653
I20260810 03:44:36.863658 13415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43653 every 8 connection(s)
I20260810 03:44:36.864517 13340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:36.867399 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 13340
I20260810 03:44:36.867566 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:36.867712 13416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:36.867931 22205 external_mini_cluster.cc:1428] Setting key f80a1870b006c6f02aa205a824189dca
I20260810 03:44:36.874122 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:36.869322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:48629 (local address 127.0.0.1:43653)
0810 03:44:36.869812 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:36.869834 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:36.869881 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:44:36.870919 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:36.870948 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:36.871362 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:36.871771 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:36.872500 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:36.872506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:36.872523 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:36.872535 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:36.872603 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:36.872631 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:36.872786 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:36.872858 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 03:44:36.873044 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":174,"threads_started":1}
I20260810 03:44:36.875825 13416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533: Bootstrap starting.
I20260810 03:44:36.876938 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:44:36.878558 13416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:36.879530 13416 log.cc:824] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:36.881006 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:36.879165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:56945 (local address 127.0.0.1:43653)
0810 03:44:36.879307 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:36.879312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:36.879449 (+ 137us) server_negotiation.cc:407] Connection header received
0810 03:44:36.879532 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:36.879535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:36.879580 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:36.879675 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:36.880572 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:36.880576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:36.880579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:36.880592 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:36.880627 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:36.880647 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:36.880757 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:36.880822 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 03:44:36.880862 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:44:36.882095 13416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533: No bootstrap required, opened a new log
I20260810 03:44:36.886037 13416 raft_consensus.cc:348] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41351887fcc43a3b9a24e9fa6a4c533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43653 } }
I20260810 03:44:36.886308 13416 raft_consensus.cc:374] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:36.886399 13416 raft_consensus.cc:729] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c41351887fcc43a3b9a24e9fa6a4c533, State: Initialized, Role: FOLLOWER
I20260810 03:44:36.886848 13416 consensus_queue.cc:260] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41351887fcc43a3b9a24e9fa6a4c533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43653 } }
I20260810 03:44:36.887027 13416 raft_consensus.cc:388] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:36.887125 13416 raft_consensus.cc:482] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:36.887257 13416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:36.888260 13416 raft_consensus.cc:504] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41351887fcc43a3b9a24e9fa6a4c533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43653 } }
I20260810 03:44:36.888635 13416 leader_election.cc:302] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c41351887fcc43a3b9a24e9fa6a4c533; no voters:
I20260810 03:44:36.888988 13416 leader_election.cc:288] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:36.889139 13427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:36.889413 13427 raft_consensus.cc:686] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [term 1 LEADER]: Becoming Leader. State: Replica: c41351887fcc43a3b9a24e9fa6a4c533, State: Running, Role: LEADER
I20260810 03:44:36.889786 13427 consensus_queue.cc:237] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41351887fcc43a3b9a24e9fa6a4c533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43653 } }
I20260810 03:44:36.890359 13416 sys_catalog.cc:564] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:36.891708 13429 sys_catalog.cc:455] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c41351887fcc43a3b9a24e9fa6a4c533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41351887fcc43a3b9a24e9fa6a4c533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43653 } } }
I20260810 03:44:36.891970 13429 sys_catalog.cc:458] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:36.892079 13430 sys_catalog.cc:455] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c41351887fcc43a3b9a24e9fa6a4c533. Latest consensus state: current_term: 1 leader_uuid: "c41351887fcc43a3b9a24e9fa6a4c533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41351887fcc43a3b9a24e9fa6a4c533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43653 } } }
I20260810 03:44:36.892315 13430 sys_catalog.cc:458] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:36.893038 13434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:36.896160 13434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:36.902045 13434 catalog_manager.cc:1347] Generated new cluster ID: 7a3635858cbd4023a7e4cf77f3ca4729
I20260810 03:44:36.902155 13434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:36.927974 13434 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:36.928342 13434 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:36.944363 13434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c41351887fcc43a3b9a24e9fa6a4c533: Generated new TSK 0
I20260810 03:44:36.945143 13434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:36.953524 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 13340
2026-08-10T03:44:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1592 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:37 dist-test-slave-ss4n krb5kdc[13449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:37 dist-test-slave-ss4n krb5kdc[13449](info): set up 2 sockets
Aug 10 03:44:37 dist-test-slave-ss4n krb5kdc[13449](info): commencing operation
krb5kdc: starting...
W20260810 03:44:38.125471 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:38 dist-test-slave-ss4n krb5kdc[13449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:44:38.262883 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:32893
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:41063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:32893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:38.381753 13463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:38.382113 13463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:38.382218 13463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:38.386312 13463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:38.386425 13463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:38.386466 13463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:38.386504 13463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:38.386539 13463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:38.392082 13463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:41063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:32893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:32893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:38.393273 13463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:38.394382 13463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:38.403736 13470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:38.403736 13472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:38.403775 13463 server_base.cc:1034] running on GCE node
W20260810 03:44:38.403733 13469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:38.404604 13463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:38.405962 13463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:38.407176 13463 hybrid_clock.cc:648] HybridClock initialized: now 1786333478407154 us; error 35 us; skew 500 ppm
Aug 10 03:44:38 dist-test-slave-ss4n krb5kdc[13449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333478, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:38.414808 13463 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:44:38.415841 13463 webserver.cc:460] Webserver started at http://127.21.175.126:46729/ using document root <none> and password file <none>
I20260810 03:44:38.416437 13463 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:38.416513 13463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:38.416702 13463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:38.419087 13463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "019d0b7ab1164c809fbc42befdb56ccc"
format_stamp: "Formatted at 2026-08-10 03:44:38 on dist-test-slave-ss4n"
server_key: "5e0f6a9e3116798ca1e3b23a12f1c745"
server_key_iv: "9a8526fac26d5756e3ecab9b028003d5"
server_key_version: "encryptionkey@0"
I20260810 03:44:38.419680 13463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "019d0b7ab1164c809fbc42befdb56ccc"
format_stamp: "Formatted at 2026-08-10 03:44:38 on dist-test-slave-ss4n"
server_key: "5e0f6a9e3116798ca1e3b23a12f1c745"
server_key_iv: "9a8526fac26d5756e3ecab9b028003d5"
server_key_version: "encryptionkey@0"
I20260810 03:44:38.424124 13463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:44:38.426872 13481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:38.428256 13463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:44:38.428416 13463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "019d0b7ab1164c809fbc42befdb56ccc"
format_stamp: "Formatted at 2026-08-10 03:44:38 on dist-test-slave-ss4n"
server_key: "5e0f6a9e3116798ca1e3b23a12f1c745"
server_key_iv: "9a8526fac26d5756e3ecab9b028003d5"
server_key_version: "encryptionkey@0"
I20260810 03:44:38.428539 13463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:38.438570 13463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:38.442016 13463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:38.442198 13463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:38.448944 13463 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:32893
I20260810 03:44:38.449021 13539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:32893 every 8 connection(s)
I20260810 03:44:38.450038 13463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:38.451120 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 13463
I20260810 03:44:38.451275 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:38.451613 22205 external_mini_cluster.cc:1428] Setting key 742540b41b3c53a68bc9981038dbed6f
I20260810 03:44:38.453276 13541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:38.459052 13541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc: Bootstrap starting.
Aug 10 03:44:38 dist-test-slave-ss4n krb5kdc[13449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:38.461854 13541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:38.462823 13541 log.cc:824] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:38.465104 13541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc: No bootstrap required, opened a new log
I20260810 03:44:38.467065 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:38.453117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49069 (local address 127.21.175.126:32893)
0810 03:44:38.453684 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:38.453701 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:38.453737 (+ 36us) server_negotiation.cc:407] Connection header received
0810 03:44:38.454370 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:38.454387 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:38.454741 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:38.455090 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:38.455950 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:38.457011 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:38.457974 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:38.458461 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:38.458667 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:38.461687 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:38.461692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:38.461706 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:38.461737 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:38.463548 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:38.464151 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:38.464156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:38.464160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:38.464269 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:38.464677 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:38.464680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:38.464681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:38.465104 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:38.465371 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:38.465779 (+ 408us) server_negotiation.cc:299] Negotiation successful
0810 03:44:38.466016 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":150,"threads_started":1}
I20260810 03:44:38.468768 13541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019d0b7ab1164c809fbc42befdb56ccc" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 32893 } }
I20260810 03:44:38.469048 13541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:38.469167 13541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019d0b7ab1164c809fbc42befdb56ccc, State: Initialized, Role: FOLLOWER
I20260810 03:44:38.469732 13541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019d0b7ab1164c809fbc42befdb56ccc" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 32893 } }
I20260810 03:44:38.469899 13541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:38.469944 13541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:38.470067 13541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:38.470430 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:44:38.471334 13541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019d0b7ab1164c809fbc42befdb56ccc" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 32893 } }
I20260810 03:44:38.471740 13541 leader_election.cc:302] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 019d0b7ab1164c809fbc42befdb56ccc; no voters:
I20260810 03:44:38.472157 13541 leader_election.cc:288] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:38.472254 13550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:38.472522 13550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [term 1 LEADER]: Becoming Leader. State: Replica: 019d0b7ab1164c809fbc42befdb56ccc, State: Running, Role: LEADER
I20260810 03:44:38.472954 13550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019d0b7ab1164c809fbc42befdb56ccc" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 32893 } }
I20260810 03:44:38.473613 13541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:38.474823 13551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "019d0b7ab1164c809fbc42befdb56ccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019d0b7ab1164c809fbc42befdb56ccc" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 32893 } } }
I20260810 03:44:38.475014 13551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:38.475384 13552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 019d0b7ab1164c809fbc42befdb56ccc. Latest consensus state: current_term: 1 leader_uuid: "019d0b7ab1164c809fbc42befdb56ccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019d0b7ab1164c809fbc42befdb56ccc" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 32893 } } }
I20260810 03:44:38.475555 13552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:38.476364 13563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:38.479522 13563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:38.483949 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:38.472811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36949 (local address 127.21.175.126:32893)
0810 03:44:38.472931 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:38.472936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:38.474477 (+ 1541us) server_negotiation.cc:407] Connection header received
0810 03:44:38.474709 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:38.474712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:38.474766 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:38.474826 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:38.475820 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:38.476503 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:38.477542 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:38.477899 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:38.477998 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:38.479950 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:38.479954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:38.479957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:38.479984 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:38.481851 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:38.482485 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:38.482489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:38.482491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:38.482586 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:38.482961 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:38.482965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:38.482967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:38.483210 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:38.483362 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:38.483696 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 03:44:38.483794 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:44:38.485419 13563 catalog_manager.cc:1347] Generated new cluster ID: cde50150dfed4ea7a265e0e1fa68d04c
I20260810 03:44:38.485503 13563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:38.493960 13563 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:38.495244 13563 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:38.504578 13563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 019d0b7ab1164c809fbc42befdb56ccc: Generated new TSK 0
I20260810 03:44:38.505420 13563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:38.513350 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 13463
W20260810 03:44:38.521783 13447 connection.cc:535] client connection to 127.21.175.126:32893 recv error: Network error: failed to read from TLS socket (remote: 127.21.175.126:32893): Connection reset by peer (error 104)
2026-08-10T03:44:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1560 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:38 dist-test-slave-ss4n krb5kdc[13574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:38 dist-test-slave-ss4n krb5kdc[13574](info): set up 2 sockets
Aug 10 03:44:38 dist-test-slave-ss4n krb5kdc[13574](info): commencing operation
krb5kdc: starting...
W20260810 03:44:39.682410 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:39 dist-test-slave-ss4n krb5kdc[13574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:44:39.821035 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:39087
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:35785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:39087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:39.938611 13589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:39.938925 13589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:39.939023 13589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:39.943169 13589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:39.943282 13589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:39.943323 13589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:39.943382 13589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:39.943420 13589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:39.948944 13589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:35785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:39087
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:39087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:39.950145 13589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:39.951203 13589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:39.960083 13597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:39.960225 13589 server_base.cc:1034] running on GCE node
W20260810 03:44:39.960151 13595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:39.960083 13594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:39.961009 13589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:39.962316 13589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:39.963559 13589 hybrid_clock.cc:648] HybridClock initialized: now 1786333479963521 us; error 48 us; skew 500 ppm
Aug 10 03:44:39 dist-test-slave-ss4n krb5kdc[13574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333479, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:39.966852 13589 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:44:39.967883 13589 webserver.cc:460] Webserver started at http://127.21.175.126:36931/ using document root <none> and password file <none>
I20260810 03:44:39.968514 13589 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:39.968619 13589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:39.968875 13589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:39.971200 13589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "f6108e5827564762b94bf7f07685c753"
format_stamp: "Formatted at 2026-08-10 03:44:39 on dist-test-slave-ss4n"
server_key: "c1628bc7e03cbd20808fe82e13672362"
server_key_iv: "047226aba94e06bf11e883e7dcc773e1"
server_key_version: "encryptionkey@0"
I20260810 03:44:39.971812 13589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "f6108e5827564762b94bf7f07685c753"
format_stamp: "Formatted at 2026-08-10 03:44:39 on dist-test-slave-ss4n"
server_key: "c1628bc7e03cbd20808fe82e13672362"
server_key_iv: "047226aba94e06bf11e883e7dcc773e1"
server_key_version: "encryptionkey@0"
I20260810 03:44:39.975835 13589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:44:39.978528 13604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:39.979768 13589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:44:39.979913 13589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "f6108e5827564762b94bf7f07685c753"
format_stamp: "Formatted at 2026-08-10 03:44:39 on dist-test-slave-ss4n"
server_key: "c1628bc7e03cbd20808fe82e13672362"
server_key_iv: "047226aba94e06bf11e883e7dcc773e1"
server_key_version: "encryptionkey@0"
I20260810 03:44:39.980028 13589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:39.997710 13589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:40.001297 13589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:40.001538 13589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:40.008682 13589 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:39087
I20260810 03:44:40.008745 13665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:39087 every 8 connection(s)
I20260810 03:44:40.009768 13589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:40.012624 13666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:40.018285 13666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753: Bootstrap starting.
I20260810 03:44:40.018675 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 13589
I20260810 03:44:40.018822 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:40.019116 22205 external_mini_cluster.cc:1428] Setting key eb48a1edca16970aaaa5c204394d0948
I20260810 03:44:40.021135 13666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:40.021943 13666 log.cc:824] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:40.023983 13666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753: No bootstrap required, opened a new log
I20260810 03:44:40.028347 13666 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6108e5827564762b94bf7f07685c753" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39087 } }
I20260810 03:44:40.028564 13666 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:40.028633 13666 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6108e5827564762b94bf7f07685c753, State: Initialized, Role: FOLLOWER
I20260810 03:44:40.028692 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:40.020507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50009 (local address 127.21.175.126:39087)
0810 03:44:40.020957 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:40.020972 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:40.021008 (+ 36us) server_negotiation.cc:407] Connection header received
0810 03:44:40.021734 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:40.021755 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:40.022168 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:40.022527 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:40.022984 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:40.023933 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:40.025140 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:40.025679 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:40.025798 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:40.026778 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:40.026783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:40.026796 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:40.027230 (+ 434us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:40.027294 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:40.027320 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:40.027447 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:40.027719 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 03:44:40.027869 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":130,"threads_started":1}
I20260810 03:44:40.029129 13666 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6108e5827564762b94bf7f07685c753" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39087 } }
I20260810 03:44:40.029258 13666 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:40.029340 13666 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:40.029465 13666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:40.030686 13666 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6108e5827564762b94bf7f07685c753" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39087 } }
I20260810 03:44:40.031172 13666 leader_election.cc:302] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6108e5827564762b94bf7f07685c753; no voters:
I20260810 03:44:40.031600 13666 leader_election.cc:288] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:40.031702 13672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:40.031955 13672 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [term 1 LEADER]: Becoming Leader. State: Replica: f6108e5827564762b94bf7f07685c753, State: Running, Role: LEADER
I20260810 03:44:40.032094 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:44:40.032342 13672 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6108e5827564762b94bf7f07685c753" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39087 } }
I20260810 03:44:40.033015 13666 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:40.033922 13674 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6108e5827564762b94bf7f07685c753. Latest consensus state: current_term: 1 leader_uuid: "f6108e5827564762b94bf7f07685c753" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6108e5827564762b94bf7f07685c753" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39087 } } }
I20260810 03:44:40.034044 13674 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:40.034377 13673 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6108e5827564762b94bf7f07685c753" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6108e5827564762b94bf7f07685c753" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39087 } } }
I20260810 03:44:40.034456 13673 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:40.034875 13689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:40.038244 13689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:40.040912 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:40.035436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33415 (local address 127.21.175.126:39087)
0810 03:44:40.035549 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:40.035554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:40.036260 (+ 706us) server_negotiation.cc:407] Connection header received
0810 03:44:40.036450 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:40.036454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:40.036500 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:40.036550 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:40.036968 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:40.037753 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:40.038922 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:40.039257 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:40.039363 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:40.040388 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:40.040392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:40.040394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:40.040408 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:40.040438 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:40.040452 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:40.040554 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:40.040740 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 03:44:40.040779 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 03:44:40.044729 13689 catalog_manager.cc:1347] Generated new cluster ID: 0906dea6147a41ba996f8b2ae45123e3
I20260810 03:44:40.044803 13689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:40.060688 13689 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:40.062124 13689 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:40.071010 13689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6108e5827564762b94bf7f07685c753: Generated new TSK 0
I20260810 03:44:40.071977 13689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:40.086050 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 13589
W20260810 03:44:40.095299 13572 connection.cc:535] client connection to 127.21.175.126:39087 recv error: Network error: failed to read from TLS socket (remote: 127.21.175.126:39087): Connection reset by peer (error 104)
2026-08-10T03:44:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1574 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:40 dist-test-slave-ss4n krb5kdc[13704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:40 dist-test-slave-ss4n krb5kdc[13704](info): set up 2 sockets
Aug 10 03:44:40 dist-test-slave-ss4n krb5kdc[13704](info): commencing operation
krb5kdc: starting...
W20260810 03:44:41.233050 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:41 dist-test-slave-ss4n krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:44:41.380854 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:44781
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:44781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:41.496002 13717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:41.496290 13717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:41.496331 13717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:41.500218 13717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:41.500273 13717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:41.500293 13717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:41.500309 13717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:41.500324 13717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:41.505646 13717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:44781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:44781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:41.506704 13717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:41.507671 13717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:41.516645 13723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:41.516687 13725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:41.516690 13722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:41.516947 13717 server_base.cc:1034] running on GCE node
I20260810 03:44:41.517720 13717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:41.518947 13717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:41.520138 13717 hybrid_clock.cc:648] HybridClock initialized: now 1786333481520105 us; error 43 us; skew 500 ppm
Aug 10 03:44:41 dist-test-slave-ss4n krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333481, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:41.523524 13717 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:44:41.524515 13717 webserver.cc:460] Webserver started at http://127.21.175.126:37755/ using document root <none> and password file <none>
I20260810 03:44:41.525136 13717 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:41.525238 13717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:41.525563 13717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:41.527822 13717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "847b534241024fee961b922501532f5b"
format_stamp: "Formatted at 2026-08-10 03:44:41 on dist-test-slave-ss4n"
server_key: "883182e273980b63289bab0be123ff5e"
server_key_iv: "85fef631c73a11a004d4e9b7b692a9d9"
server_key_version: "encryptionkey@0"
I20260810 03:44:41.528460 13717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "847b534241024fee961b922501532f5b"
format_stamp: "Formatted at 2026-08-10 03:44:41 on dist-test-slave-ss4n"
server_key: "883182e273980b63289bab0be123ff5e"
server_key_iv: "85fef631c73a11a004d4e9b7b692a9d9"
server_key_version: "encryptionkey@0"
I20260810 03:44:41.532559 13717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 03:44:41.535306 13732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:41.536840 13717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:44:41.537021 13717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "847b534241024fee961b922501532f5b"
format_stamp: "Formatted at 2026-08-10 03:44:41 on dist-test-slave-ss4n"
server_key: "883182e273980b63289bab0be123ff5e"
server_key_iv: "85fef631c73a11a004d4e9b7b692a9d9"
server_key_version: "encryptionkey@0"
I20260810 03:44:41.537156 13717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:41.542047 13717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:41.545588 13717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:41.545820 13717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:41.553943 13717 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:44781
I20260810 03:44:41.554044 13789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:44781 every 8 connection(s)
I20260810 03:44:41.555104 13717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:41.558303 13790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:41.558817 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 13717
I20260810 03:44:41.558974 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:41.559298 22205 external_mini_cluster.cc:1428] Setting key a21ba8c859b2214902b18121cb09d574
I20260810 03:44:41.565062 13790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b: Bootstrap starting.
I20260810 03:44:41.565109 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:41.560812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60359 (local address 127.21.175.126:44781)
0810 03:44:41.561274 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:41.561291 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:41.561329 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:44:41.562102 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:41.562121 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:41.562442 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:41.562747 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:41.563711 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:41.563716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:41.563729 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:41.563742 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:41.563804 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:41.563826 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:41.563976 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:41.564099 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 03:44:41.564296 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":154,"threads_started":1}
I20260810 03:44:41.567646 13790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:41.568372 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:44:41.568588 13790 log.cc:824] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:41.571019 13790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b: No bootstrap required, opened a new log
I20260810 03:44:41.572710 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:41.570689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47297 (local address 127.21.175.126:44781)
0810 03:44:41.570821 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:41.570825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:41.571026 (+ 201us) server_negotiation.cc:407] Connection header received
0810 03:44:41.571155 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:41.571158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:41.571193 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:41.571283 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:41.572212 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:41.572216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:41.572219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:41.572230 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:41.572259 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:41.572272 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:41.572361 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:41.572553 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 03:44:41.572591 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:44:41.575421 13790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847b534241024fee961b922501532f5b" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44781 } }
I20260810 03:44:41.575695 13790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:41.575755 13790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847b534241024fee961b922501532f5b, State: Initialized, Role: FOLLOWER
I20260810 03:44:41.576182 13790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847b534241024fee961b922501532f5b" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44781 } }
I20260810 03:44:41.576344 13790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:41.576436 13790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:41.576542 13790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:41.577579 13790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847b534241024fee961b922501532f5b" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44781 } }
I20260810 03:44:41.577939 13790 leader_election.cc:302] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 847b534241024fee961b922501532f5b; no voters:
I20260810 03:44:41.578277 13790 leader_election.cc:288] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:41.578418 13801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:41.578655 13801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [term 1 LEADER]: Becoming Leader. State: Replica: 847b534241024fee961b922501532f5b, State: Running, Role: LEADER
I20260810 03:44:41.579041 13801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847b534241024fee961b922501532f5b" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44781 } }
I20260810 03:44:41.579663 13790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:41.580714 13803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 847b534241024fee961b922501532f5b. Latest consensus state: current_term: 1 leader_uuid: "847b534241024fee961b922501532f5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847b534241024fee961b922501532f5b" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44781 } } }
I20260810 03:44:41.580907 13803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:41.581262 13802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "847b534241024fee961b922501532f5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847b534241024fee961b922501532f5b" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44781 } } }
I20260810 03:44:41.581341 13810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:41.581429 13802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:41.584271 13810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:41.589840 13810 catalog_manager.cc:1347] Generated new cluster ID: 430724266b384bcfa70ede8f69c2560b
I20260810 03:44:41.589920 13810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:41.615175 13810 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:41.615541 13810 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:41.623289 13810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 847b534241024fee961b922501532f5b: Generated new TSK 0
I20260810 03:44:41.624119 13810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:41.639504 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 13717
2026-08-10T03:44:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1550 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:41 dist-test-slave-ss4n krb5kdc[13824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:41 dist-test-slave-ss4n krb5kdc[13824](info): set up 2 sockets
Aug 10 03:44:41 dist-test-slave-ss4n krb5kdc[13824](info): commencing operation
krb5kdc: starting...
W20260810 03:44:42.840530 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:42 dist-test-slave-ss4n krb5kdc[13824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:44:42.975338 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:39553
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:39553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:43.091337 13837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:43.091805 13837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:43.091871 13837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:43.096021 13837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:43.096096 13837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:43.096115 13837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:43.096132 13837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:43.096148 13837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:43.101785 13837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:39553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:39553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:43.102910 13837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:43.103941 13837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:43.113131 13847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:43.113121 13845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:43.113123 13844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:43.113534 13837 server_base.cc:1034] running on GCE node
I20260810 03:44:43.114130 13837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:43.115249 13837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:43.116420 13837 hybrid_clock.cc:648] HybridClock initialized: now 1786333483116404 us; error 31 us; skew 500 ppm
Aug 10 03:44:43 dist-test-slave-ss4n krb5kdc[13824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333483, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:43.119304 13837 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:44:43.120563 13837 webserver.cc:460] Webserver started at http://127.21.175.126:33445/ using document root <none> and password file <none>
I20260810 03:44:43.121292 13837 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:43.121384 13837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:43.121690 13837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:43.124248 13837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "a90788300f87456e872b31ce554b08fb"
format_stamp: "Formatted at 2026-08-10 03:44:43 on dist-test-slave-ss4n"
server_key: "4ff3b2a2e091693c37742726e61cdfa2"
server_key_iv: "1d6f00f982fad0e1a322c37627e48d7f"
server_key_version: "encryptionkey@0"
I20260810 03:44:43.124854 13837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "a90788300f87456e872b31ce554b08fb"
format_stamp: "Formatted at 2026-08-10 03:44:43 on dist-test-slave-ss4n"
server_key: "4ff3b2a2e091693c37742726e61cdfa2"
server_key_iv: "1d6f00f982fad0e1a322c37627e48d7f"
server_key_version: "encryptionkey@0"
I20260810 03:44:43.128756 13837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 03:44:43.131863 13856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:43.133365 13837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:44:43.133546 13837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "a90788300f87456e872b31ce554b08fb"
format_stamp: "Formatted at 2026-08-10 03:44:43 on dist-test-slave-ss4n"
server_key: "4ff3b2a2e091693c37742726e61cdfa2"
server_key_iv: "1d6f00f982fad0e1a322c37627e48d7f"
server_key_version: "encryptionkey@0"
I20260810 03:44:43.133674 13837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:43.155818 13837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:43.159613 13837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:43.159848 13837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:43.168246 13837 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:39553
I20260810 03:44:43.168285 13913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:39553 every 8 connection(s)
I20260810 03:44:43.169267 13837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:43.172201 13914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:43.173184 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 13837
I20260810 03:44:43.173352 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:43.173704 22205 external_mini_cluster.cc:1428] Setting key 65d99888cabb43161d5e0d0ccc36f588
I20260810 03:44:43.179093 13914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb: Bootstrap starting.
I20260810 03:44:43.181808 13914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:43.182639 13914 log.cc:824] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:44:43 dist-test-slave-ss4n krb5kdc[13824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:43.185075 13914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb: No bootstrap required, opened a new log
I20260810 03:44:43.190114 13914 raft_consensus.cc:348] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90788300f87456e872b31ce554b08fb" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39553 } }
I20260810 03:44:43.190394 13914 raft_consensus.cc:374] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:43.190445 13914 raft_consensus.cc:729] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90788300f87456e872b31ce554b08fb, State: Initialized, Role: FOLLOWER
I20260810 03:44:43.190721 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:43.175222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38451 (local address 127.21.175.126:39553)
0810 03:44:43.175671 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:43.175690 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:43.175735 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:44:43.176577 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:43.176602 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:43.177040 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:43.177434 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:43.178268 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:43.179678 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:43.180762 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:43.181436 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:43.181650 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:43.185086 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:43.185096 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:43.185116 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:43.185152 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:43.187354 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:43.187833 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:43.187840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:43.187846 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:43.187994 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:43.188263 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:43.188266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:43.188268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:43.188799 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:43.189003 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:43.189359 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 03:44:43.189709 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":117,"threads_started":1}
I20260810 03:44:43.190949 13914 consensus_queue.cc:260] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90788300f87456e872b31ce554b08fb" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39553 } }
I20260810 03:44:43.191113 13914 raft_consensus.cc:388] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:43.191181 13914 raft_consensus.cc:482] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:43.191279 13914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:43.192229 13914 raft_consensus.cc:504] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90788300f87456e872b31ce554b08fb" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39553 } }
I20260810 03:44:43.192592 13914 leader_election.cc:302] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a90788300f87456e872b31ce554b08fb; no voters:
I20260810 03:44:43.192945 13914 leader_election.cc:288] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:43.193135 13919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:43.193426 13919 raft_consensus.cc:686] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [term 1 LEADER]: Becoming Leader. State: Replica: a90788300f87456e872b31ce554b08fb, State: Running, Role: LEADER
I20260810 03:44:43.193827 13919 consensus_queue.cc:237] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90788300f87456e872b31ce554b08fb" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39553 } }
I20260810 03:44:43.193944 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:44:43.194357 13914 sys_catalog.cc:564] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:43.195632 13920 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a90788300f87456e872b31ce554b08fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90788300f87456e872b31ce554b08fb" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39553 } } }
I20260810 03:44:43.195596 13922 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a90788300f87456e872b31ce554b08fb. Latest consensus state: current_term: 1 leader_uuid: "a90788300f87456e872b31ce554b08fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90788300f87456e872b31ce554b08fb" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 39553 } } }
I20260810 03:44:43.195740 13922 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:43.195740 13920 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:43.196417 13934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:43.200083 13934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:43.205828 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:43.196240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44787 (local address 127.21.175.126:39553)
0810 03:44:43.196356 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:43.196361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:43.196803 (+ 442us) server_negotiation.cc:407] Connection header received
0810 03:44:43.197015 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:43.197019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:43.197088 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:43.197150 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:43.198058 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:43.198816 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:43.199959 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:43.200278 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:43.200388 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:43.202198 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:43.202202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:43.202205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:43.202232 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:43.203951 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:43.204426 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:43.204431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:43.204433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:43.204539 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:43.204864 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:43.204867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:43.204869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:43.205103 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:43.205232 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:43.205557 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 03:44:43.205684 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 03:44:43.206187 13934 catalog_manager.cc:1347] Generated new cluster ID: 255784a76e1a439faf869188ec4fd5e4
I20260810 03:44:43.206272 13934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:43.249745 13934 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:43.251129 13934 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:43.257892 13934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a90788300f87456e872b31ce554b08fb: Generated new TSK 0
I20260810 03:44:43.258627 13934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:43.268931 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 13837
2026-08-10T03:44:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1630 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:43 dist-test-slave-ss4n krb5kdc[13948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:43 dist-test-slave-ss4n krb5kdc[13948](info): set up 2 sockets
Aug 10 03:44:43 dist-test-slave-ss4n krb5kdc[13948](info): commencing operation
krb5kdc: starting...
W20260810 03:44:44.377159 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:44 dist-test-slave-ss4n krb5kdc[13948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:44:44.517974 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:34425
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:34425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:44.634086 13961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:44.634390 13961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:44.634435 13961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:44.638600 13961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:44.638684 13961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:44.638705 13961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:44.638721 13961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:44.638737 13961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:44.644382 13961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:34425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:34425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:44.645956 13961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:44.646992 13961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:44.656013 13968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:44.656008 13970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:44.656049 13967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:44.656281 13961 server_base.cc:1034] running on GCE node
I20260810 03:44:44.656862 13961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:44.658129 13961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:44.659330 13961 hybrid_clock.cc:648] HybridClock initialized: now 1786333484659311 us; error 32 us; skew 500 ppm
Aug 10 03:44:44 dist-test-slave-ss4n krb5kdc[13948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333484, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:44.662323 13961 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:44:44.663208 13961 webserver.cc:460] Webserver started at http://127.21.175.126:36347/ using document root <none> and password file <none>
I20260810 03:44:44.663780 13961 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:44.663842 13961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:44.664011 13961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:44.666153 13961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "8aac793df02b40dfa8a59254c378f703"
format_stamp: "Formatted at 2026-08-10 03:44:44 on dist-test-slave-ss4n"
server_key: "ba5eb14bfec244895a2ba54ca762bbcb"
server_key_iv: "016cad94d80d17fcb283fa42fd4e2f7b"
server_key_version: "encryptionkey@0"
I20260810 03:44:44.666685 13961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "8aac793df02b40dfa8a59254c378f703"
format_stamp: "Formatted at 2026-08-10 03:44:44 on dist-test-slave-ss4n"
server_key: "ba5eb14bfec244895a2ba54ca762bbcb"
server_key_iv: "016cad94d80d17fcb283fa42fd4e2f7b"
server_key_version: "encryptionkey@0"
I20260810 03:44:44.671087 13961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:44:44.673882 13978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:44.675264 13961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:44:44.675379 13961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "8aac793df02b40dfa8a59254c378f703"
format_stamp: "Formatted at 2026-08-10 03:44:44 on dist-test-slave-ss4n"
server_key: "ba5eb14bfec244895a2ba54ca762bbcb"
server_key_iv: "016cad94d80d17fcb283fa42fd4e2f7b"
server_key_version: "encryptionkey@0"
I20260810 03:44:44.675532 13961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:44.711580 13961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:44.715482 13961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:44.715713 13961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:44.722946 13961 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:34425
I20260810 03:44:44.722960 14037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:34425 every 8 connection(s)
I20260810 03:44:44.723960 13961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:44.726554 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 13961
I20260810 03:44:44.726699 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:44.727027 22205 external_mini_cluster.cc:1428] Setting key 90749b61d4e86ea370018f668d4891e1
I20260810 03:44:44.727123 14039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:44.734521 14039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703: Bootstrap starting.
I20260810 03:44:44.736984 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:44.728551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60571 (local address 127.21.175.126:34425)
0810 03:44:44.728971 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:44.728987 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:44.729024 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:44:44.729961 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:44.729981 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:44.730328 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:44.730618 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:44.730984 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:44.731973 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:44.733228 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:44.733789 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:44.733917 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:44.734870 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:44.734876 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:44.734892 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:44.735390 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:44.735456 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:44.735489 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:44.735588 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:44.735760 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 03:44:44.736044 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":141,"threads_started":1}
I20260810 03:44:44.737434 14039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:44.738360 14039 log.cc:824] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:44.740401 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:44:44.740765 14039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703: No bootstrap required, opened a new log
I20260810 03:44:44.745146 14039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aac793df02b40dfa8a59254c378f703" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 34425 } }
I20260810 03:44:44.745393 14039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:44.745430 14039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aac793df02b40dfa8a59254c378f703, State: Initialized, Role: FOLLOWER
I20260810 03:44:44.745998 14039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aac793df02b40dfa8a59254c378f703" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 34425 } }
I20260810 03:44:44.746177 14039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:44.746299 14039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:44.746425 14039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:44.747602 14039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aac793df02b40dfa8a59254c378f703" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 34425 } }
I20260810 03:44:44.747700 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:44.742930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33649 (local address 127.21.175.126:34425)
0810 03:44:44.743080 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:44.743084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:44.743298 (+ 214us) server_negotiation.cc:407] Connection header received
0810 03:44:44.743395 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:44.743399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:44.743457 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:44.743576 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:44.743832 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:44.744636 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:44.745697 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:44.746037 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:44.746161 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:44.747209 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:44.747216 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:44.747223 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:44.747241 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:44.747273 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:44.747286 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:44.747371 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:44.747453 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 03:44:44.747514 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:44:44.748018 14039 leader_election.cc:302] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8aac793df02b40dfa8a59254c378f703; no voters:
I20260810 03:44:44.748382 14039 leader_election.cc:288] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:44.748515 14050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:44.748857 14050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [term 1 LEADER]: Becoming Leader. State: Replica: 8aac793df02b40dfa8a59254c378f703, State: Running, Role: LEADER
I20260810 03:44:44.749267 14050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aac793df02b40dfa8a59254c378f703" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 34425 } }
I20260810 03:44:44.750033 14039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:44.751461 14051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8aac793df02b40dfa8a59254c378f703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aac793df02b40dfa8a59254c378f703" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 34425 } } }
I20260810 03:44:44.751463 14052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8aac793df02b40dfa8a59254c378f703. Latest consensus state: current_term: 1 leader_uuid: "8aac793df02b40dfa8a59254c378f703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aac793df02b40dfa8a59254c378f703" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 34425 } } }
I20260810 03:44:44.751638 14052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:44.751639 14051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:44.752079 14060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:44.755833 14060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:44.762504 14060 catalog_manager.cc:1347] Generated new cluster ID: d23752e02f724980ba869e473c912d9e
I20260810 03:44:44.762602 14060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:44.772286 14060 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:44.773321 14060 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:44.784431 14060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8aac793df02b40dfa8a59254c378f703: Generated new TSK 0
I20260810 03:44:44.785277 14060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:44.795687 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 13961
2026-08-10T03:44:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1528 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:44 dist-test-slave-ss4n krb5kdc[14073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:44 dist-test-slave-ss4n krb5kdc[14073](info): set up 2 sockets
Aug 10 03:44:44 dist-test-slave-ss4n krb5kdc[14073](info): commencing operation
krb5kdc: starting...
W20260810 03:44:45.982180 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:46 dist-test-slave-ss4n krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:44:46.117724 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:34677
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:40089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:34677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:46.231334 14088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:46.231628 14088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:46.231673 14088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:46.235713 14088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:46.235772 14088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:46.235792 14088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:46.235808 14088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:46.235872 14088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:46.241106 14088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:40089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:34677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:34677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:46.242264 14088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:46.243397 14088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:46.252172 14094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:46.252173 14093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:46.252156 14096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:46.252341 14088 server_base.cc:1034] running on GCE node
I20260810 03:44:46.253052 14088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:46.254292 14088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:46.255460 14088 hybrid_clock.cc:648] HybridClock initialized: now 1786333486255445 us; error 28 us; skew 500 ppm
Aug 10 03:44:46 dist-test-slave-ss4n krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333486, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:46.258389 14088 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:44:46.259322 14088 webserver.cc:460] Webserver started at http://127.21.175.126:46483/ using document root <none> and password file <none>
I20260810 03:44:46.259879 14088 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:46.259940 14088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:46.260157 14088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:46.262303 14088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "ae54edfddcd34572a46e5cac6c9055c8"
format_stamp: "Formatted at 2026-08-10 03:44:46 on dist-test-slave-ss4n"
server_key: "e2294e224e100edd332ea6b533ad2498"
server_key_iv: "a058d4623cd282db45f9c403703f9c00"
server_key_version: "encryptionkey@0"
I20260810 03:44:46.262874 14088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "ae54edfddcd34572a46e5cac6c9055c8"
format_stamp: "Formatted at 2026-08-10 03:44:46 on dist-test-slave-ss4n"
server_key: "e2294e224e100edd332ea6b533ad2498"
server_key_iv: "a058d4623cd282db45f9c403703f9c00"
server_key_version: "encryptionkey@0"
I20260810 03:44:46.266685 14088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:44:46.269109 14103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:46.270432 14088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:44:46.270571 14088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "ae54edfddcd34572a46e5cac6c9055c8"
format_stamp: "Formatted at 2026-08-10 03:44:46 on dist-test-slave-ss4n"
server_key: "e2294e224e100edd332ea6b533ad2498"
server_key_iv: "a058d4623cd282db45f9c403703f9c00"
server_key_version: "encryptionkey@0"
I20260810 03:44:46.270687 14088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:46.275520 14088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:46.278704 14088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:46.278900 14088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:46.286127 14088 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:34677
I20260810 03:44:46.286118 14162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:34677 every 8 connection(s)
I20260810 03:44:46.287122 14088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:46.290153 14163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:46.294534 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 14088
I20260810 03:44:46.294693 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:46.294962 22205 external_mini_cluster.cc:1428] Setting key c8036408643a24f719048c9f19870eb2
I20260810 03:44:46.296536 14163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8: Bootstrap starting.
I20260810 03:44:46.299043 14163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:46.300014 14163 log.cc:824] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:46.300917 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:46.296511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37287 (local address 127.21.175.126:34677)
0810 03:44:46.297038 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:46.297056 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:46.297096 (+ 40us) server_negotiation.cc:407] Connection header received
0810 03:44:46.297843 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:46.297864 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:46.298165 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:46.298465 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:46.299448 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:46.299453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:46.299466 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:46.299478 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:46.299545 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:46.299567 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:46.299756 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:46.299848 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 03:44:46.300072 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":165,"threads_started":1}
I20260810 03:44:46.302232 14163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8: No bootstrap required, opened a new log
I20260810 03:44:46.304122 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:44:46.306129 14163 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae54edfddcd34572a46e5cac6c9055c8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 34677 } }
I20260810 03:44:46.306387 14163 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:44:46.306515 14163 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae54edfddcd34572a46e5cac6c9055c8, State: Initialized, Role: FOLLOWER
I20260810 03:44:46.307075 14163 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae54edfddcd34572a46e5cac6c9055c8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 34677 } }
I20260810 03:44:46.307258 14163 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:44:46.307339 14163 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:44:46.307418 14163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:46.308290 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:46.306543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54703 (local address 127.21.175.126:34677)
0810 03:44:46.306678 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:46.306681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:46.306771 (+ 90us) server_negotiation.cc:407] Connection header received
0810 03:44:46.306904 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:46.306910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:46.306957 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:46.307032 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:46.307870 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:46.307873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:46.307874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:44:46.307885 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:46.307910 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:44:46.307920 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:46.307996 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:46.308103 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 03:44:46.308134 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:44:46.308688 14163 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae54edfddcd34572a46e5cac6c9055c8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 34677 } }
I20260810 03:44:46.309062 14163 leader_election.cc:302] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae54edfddcd34572a46e5cac6c9055c8; no voters:
I20260810 03:44:46.309372 14163 leader_election.cc:288] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:44:46.309515 14173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:46.309762 14173 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [term 1 LEADER]: Becoming Leader. State: Replica: ae54edfddcd34572a46e5cac6c9055c8, State: Running, Role: LEADER
I20260810 03:44:46.310118 14173 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae54edfddcd34572a46e5cac6c9055c8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 34677 } }
I20260810 03:44:46.310704 14163 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:46.311901 14175 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae54edfddcd34572a46e5cac6c9055c8. Latest consensus state: current_term: 1 leader_uuid: "ae54edfddcd34572a46e5cac6c9055c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae54edfddcd34572a46e5cac6c9055c8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 34677 } } }
I20260810 03:44:46.311928 14174 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae54edfddcd34572a46e5cac6c9055c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae54edfddcd34572a46e5cac6c9055c8" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 34677 } } }
I20260810 03:44:46.312080 14174 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:46.312374 14175 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:46.312469 14183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:46.315454 14183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:46.321219 14183 catalog_manager.cc:1347] Generated new cluster ID: 201aaef4267148948b755dd6438ddb45
I20260810 03:44:46.321293 14183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:46.335391 14183 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:46.335754 14183 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:46.340914 14183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae54edfddcd34572a46e5cac6c9055c8: Generated new TSK 0
I20260810 03:44:46.341840 14183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:44:46.356679 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 14088
2026-08-10T03:44:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1559 ms)
[----------] 18 tests from AuthTokenIssuingTest (27972 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:46 dist-test-slave-ss4n krb5kdc[14196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:46 dist-test-slave-ss4n krb5kdc[14196](info): set up 2 sockets
Aug 10 03:44:46 dist-test-slave-ss4n krb5kdc[14196](info): commencing operation
krb5kdc: starting...
W20260810 03:44:47.469770 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:47 dist-test-slave-ss4n krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:44:47.613229 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:37991
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:37991,127.21.175.125:41175,127.21.175.124:38751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:47.729657 14211 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:44:47.730031 14211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:47.730082 14211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:47.730191 14211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:47.734279 14211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:47.734392 14211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:47.734429 14211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:47.734468 14211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:47.734503 14211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:47.739955 14211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:37991,127.21.175.125:41175,127.21.175.124:38751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:37991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:47.741173 14211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:47.742317 14211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:47.751639 14217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:47.751706 14211 server_base.cc:1034] running on GCE node
W20260810 03:44:47.751641 14221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:47.751641 14219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:47.752511 14211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:47.753733 14211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:47.754928 14211 hybrid_clock.cc:648] HybridClock initialized: now 1786333487754910 us; error 33 us; skew 500 ppm
Aug 10 03:44:47 dist-test-slave-ss4n krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:47.757992 14211 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:44:47.759039 14211 webserver.cc:460] Webserver started at http://127.21.175.126:36223/ using document root <none> and password file <none>
I20260810 03:44:47.759707 14211 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:47.759816 14211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:47.760080 14211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:47.762434 14211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "9ca202b1810041fabb3a83a50a253f70"
format_stamp: "Formatted at 2026-08-10 03:44:47 on dist-test-slave-ss4n"
server_key: "ee4cfd3f5d3862fa4964ec9247940890"
server_key_iv: "d0a69dda95a6345234f0f6d30d1e2ace"
server_key_version: "encryptionkey@0"
I20260810 03:44:47.763069 14211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "9ca202b1810041fabb3a83a50a253f70"
format_stamp: "Formatted at 2026-08-10 03:44:47 on dist-test-slave-ss4n"
server_key: "ee4cfd3f5d3862fa4964ec9247940890"
server_key_iv: "d0a69dda95a6345234f0f6d30d1e2ace"
server_key_version: "encryptionkey@0"
I20260810 03:44:47.767130 14211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:44:47.770040 14229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:47.771332 14211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:44:47.771493 14211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "9ca202b1810041fabb3a83a50a253f70"
format_stamp: "Formatted at 2026-08-10 03:44:47 on dist-test-slave-ss4n"
server_key: "ee4cfd3f5d3862fa4964ec9247940890"
server_key_iv: "d0a69dda95a6345234f0f6d30d1e2ace"
server_key_version: "encryptionkey@0"
I20260810 03:44:47.771664 14211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:47.795979 14211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:47.799180 14211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:47.799408 14211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:47.806980 14211 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:37991
I20260810 03:44:47.807291 14283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:37991 every 8 connection(s)
I20260810 03:44:47.808054 14211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:47.811020 14284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:47.811625 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 14211
I20260810 03:44:47.811769 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:47.812041 22205 external_mini_cluster.cc:1428] Setting key c466d715771248d0634ec6b86dbe22ba
I20260810 03:44:47.815212 14284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.175.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.125@KRBTEST.COM" created.
Aug 10 03:44:47 dist-test-slave-ss4n krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:47.831736 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:47.818868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.21.175.126:37991)
0810 03:44:47.819178 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:47.819194 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:47.819239 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:44:47.819717 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:47.819721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:47.819900 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:47.820098 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:47.820965 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:47.821946 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:47.823133 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:47.823591 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:47.823776 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:47.826495 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:47.826507 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:47.826522 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:47.826551 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:47.828307 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:47.828924 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:47.828929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:47.828930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:47.829066 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:47.829552 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:47.829557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:47.829559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:47.830004 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:47.830134 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:47.830662 (+ 528us) server_negotiation.cc:299] Negotiation successful
0810 03:44:47.830918 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":209,"thread_start_us":101,"threads_started":1}
I20260810 03:44:47.831733 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:47.817817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:37991 (local address 127.0.0.1:48892)
0810 03:44:47.818378 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:47.818422 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:47.819402 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:47.820092 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:47.820097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:47.820525 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:47.820788 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:47.820801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:47.822098 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:47.822103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:47.823004 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:47.823021 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:47.823746 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:47.823750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:47.823870 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:47.824493 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:47.824509 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:47.826321 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:47.828426 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:47.828437 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:47.828450 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:47.828797 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:47.829173 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:47.829179 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:47.829183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:47.829350 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:47.830144 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:47.830153 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:47.830348 (+ 195us) client_negotiation.cc:769] Sending connection context
0810 03:44:47.830549 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 03:44:47.831024 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":100,"thread_start_us":151,"threads_started":1}
I20260810 03:44:47.834632 14284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:44:47.835423 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:47.835170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:41175 (local address 127.0.0.1:46146)
0810 03:44:47.835292 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:47.835363 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:41175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 03:44:47.835678 14234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.175.125:41175: connect: Connection refused (error 111)
I20260810 03:44:47.836483 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:47.836310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:41175 (local address 127.0.0.1:46156)
0810 03:44:47.836388 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:47.836432 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:41175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260810 03:44:47.836639 14284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.125:41175: Network error: Client connection negotiation failed: client connection to 127.21.175.125:41175: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.125.keytab.
Entry for principal kudu/127.21.175.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.125.keytab.
I20260810 03:44:47.842216 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.125:41175
--webserver_interface=127.21.175.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.125.keytab
--principal=kudu/127.21.175.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:37991,127.21.175.125:41175,127.21.175.124:38751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:44:47.885913 14284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } attempt: 1
I20260810 03:44:47.886848 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:47.886533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:41175 (local address 127.0.0.1:46164)
0810 03:44:47.886688 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:47.886760 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:41175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 03:44:47.887768 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:47.887543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:41175 (local address 127.0.0.1:46172)
0810 03:44:47.887656 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:47.887705 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:41175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 03:44:47.887936 14284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.125:41175: Network error: Client connection negotiation failed: client connection to 127.21.175.125:41175: connect: Connection refused (error 111)
I20260810 03:44:47.956137 14284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } attempt: 2
I20260810 03:44:47.957206 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:47.956881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:41175 (local address 127.0.0.1:46176)
0810 03:44:47.957062 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:47.957129 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:41175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 03:44:47.958096 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:47.957836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:41175 (local address 127.0.0.1:46190)
0810 03:44:47.957982 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:47.958029 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:41175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260810 03:44:47.958236 14284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.125:41175: Network error: Client connection negotiation failed: client connection to 127.21.175.125:41175: connect: Connection refused (error 111)
W20260810 03:44:47.962744 14292 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:44:47.963027 14292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:47.963119 14292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:47.963207 14292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:47.967391 14292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:47.967453 14292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:47.967473 14292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:47.967490 14292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:47.967538 14292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:47.972970 14292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:37991,127.21.175.125:41175,127.21.175.124:38751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.125.keytab
--principal=kudu/127.21.175.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.125:41175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.175.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:47.974146 14292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:47.975268 14292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:44:47.984648 14292 server_base.cc:1034] running on GCE node
W20260810 03:44:47.984594 14300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:47.984594 14299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:47.984583 14302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:47.985415 14292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:47.986651 14292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:47.987826 14292 hybrid_clock.cc:648] HybridClock initialized: now 1786333487987804 us; error 35 us; skew 500 ppm
Aug 10 03:44:47 dist-test-slave-ss4n krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:47.990975 14292 init.cc:377] Logged in from keytab as kudu/127.21.175.125@KRBTEST.COM (short username kudu)
I20260810 03:44:47.991966 14292 webserver.cc:460] Webserver started at http://127.21.175.125:40281/ using document root <none> and password file <none>
I20260810 03:44:47.992635 14292 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:47.992735 14292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:47.992980 14292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:47.995222 14292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-1/data/instance:
uuid: "41311b868fe0469eac82046d4a9d1a33"
format_stamp: "Formatted at 2026-08-10 03:44:47 on dist-test-slave-ss4n"
server_key: "f67b4260c4831a5850dfb69d863cb3b3"
server_key_iv: "1815343c571fb84a4c78b2a36e33872d"
server_key_version: "encryptionkey@0"
I20260810 03:44:47.995823 14292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-1/wal/instance:
uuid: "41311b868fe0469eac82046d4a9d1a33"
format_stamp: "Formatted at 2026-08-10 03:44:47 on dist-test-slave-ss4n"
server_key: "f67b4260c4831a5850dfb69d863cb3b3"
server_key_iv: "1815343c571fb84a4c78b2a36e33872d"
server_key_version: "encryptionkey@0"
I20260810 03:44:47.999918 14292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:44:48.002491 14310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:48.003787 14292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:44:48.003949 14292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-1/wal
uuid: "41311b868fe0469eac82046d4a9d1a33"
format_stamp: "Formatted at 2026-08-10 03:44:47 on dist-test-slave-ss4n"
server_key: "f67b4260c4831a5850dfb69d863cb3b3"
server_key_iv: "1815343c571fb84a4c78b2a36e33872d"
server_key_version: "encryptionkey@0"
I20260810 03:44:48.004076 14292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:48.018409 14292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:48.021733 14292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:48.021982 14292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:48.029783 14292 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.125:41175
I20260810 03:44:48.030071 14366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.125:41175 every 8 connection(s)
I20260810 03:44:48.030880 14292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-1/data/info.pb
I20260810 03:44:48.033842 14367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:48.037698 14367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:44:48.040192 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 14292
I20260810 03:44:48.040344 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-1/wal/instance
I20260810 03:44:48.040648 22205 external_mini_cluster.cc:1428] Setting key dc51684aeea930727af59cb7ac169999
Aug 10 03:44:48 dist-test-slave-ss4n krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.125@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:48.049436 14284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } attempt: 3
WARNING: no policy specified for kudu/127.21.175.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.124@KRBTEST.COM" created.
I20260810 03:44:48.055220 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.039727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.21.175.126:37991)
0810 03:44:48.039966 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.039973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.040888 (+ 915us) server_negotiation.cc:407] Connection header received
0810 03:44:48.041969 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.041975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.042045 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.042174 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.043410 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.043961 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.045815 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.046069 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.046242 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.049381 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:48.049407 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:48.049413 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:48.049449 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:48.052376 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.052928 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.052932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.052934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.053027 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.053561 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.053565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.053566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.053759 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:48.053878 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:48.054894 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0810 03:44:48.055044 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
Aug 10 03:44:48 dist-test-slave-ss4n krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for kudu/127.21.175.125@KRBTEST.COM
I20260810 03:44:48.056120 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.040112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:37991 (local address 127.0.0.1:48906)
0810 03:44:48.040700 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.040756 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:48.041739 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:48.042327 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:48.042334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:48.042736 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:48.043238 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:48.043252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.044111 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.044118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:48.045651 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:48.045669 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.046310 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.046317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:48.046589 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.047300 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:48.047329 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:48.049173 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:48.052492 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:48.052499 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:48.052518 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:48.052828 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:48.053186 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:48.053189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:48.053190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:48.053415 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:48.053875 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:48.053881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:48.054300 (+ 419us) client_negotiation.cc:769] Sending connection context
0810 03:44:48.054536 (+ 236us) client_negotiation.cc:240] Negotiation successful
0810 03:44:48.054923 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"spinlock_wait_cycles":22656,"thread_start_us":195,"threads_started":1}
I20260810 03:44:48.057997 14367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:44:48.060640 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.049984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:41175 (local address 127.0.0.1:46196)
0810 03:44:48.050189 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.050205 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:48.050297 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:48.051091 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:48.051096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:48.051340 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:48.051510 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:48.051517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.052601 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.052604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:48.053328 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:48.053336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.054161 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.054168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:48.054318 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.054828 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:48.054844 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:48.056324 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:48.058761 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:48.058767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:48.058770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:48.059034 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:48.059390 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:48.059395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:48.059399 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:48.059505 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:48.060094 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:48.060097 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:48.060196 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 03:44:48.060295 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 03:44:48.060457 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260810 03:44:48.061254 14372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.050133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46196 (local address 127.21.175.125:41175)
0810 03:44:48.050528 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.050544 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.050582 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:44:48.050627 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.050634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.050877 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.051084 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.051636 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.052477 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.053461 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.054031 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.054140 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.056485 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:48.056495 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:48.056508 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:48.056534 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:48.058650 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.059159 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.059170 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.059171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.059277 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.059616 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.059621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.059626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.059915 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:48.060046 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:48.060967 (+ 921us) server_negotiation.cc:299] Negotiation successful
0810 03:44:48.061118 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":138,"threads_started":1}
Aug 10 03:44:48 dist-test-slave-ss4n krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.125@KRBTEST.COM for kudu/127.21.175.125@KRBTEST.COM
I20260810 03:44:48.064028 14284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:44:48.064777 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.064531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:38751 (local address 127.0.0.1:55632)
0810 03:44:48.064657 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.064710 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 03:44:48.065650 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.065278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:38751 (local address 127.0.0.1:55648)
0810 03:44:48.065536 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.065588 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260810 03:44:48.065841 14284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:38751: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
I20260810 03:44:48.067991 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.058563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:41175 (local address 127.0.0.1:46212)
0810 03:44:48.058882 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.058905 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:48.058998 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:48.059277 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:48.059280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:48.059463 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:48.059588 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:48.059598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.060641 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.060648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:48.061646 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:48.061663 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.062308 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.062315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:48.062433 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.062923 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:48.062939 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:48.064366 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:48.066580 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:48.066583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:48.066584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:48.066769 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:48.067088 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:48.067090 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:48.067091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:48.067165 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:48.067555 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:48.067558 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:48.067641 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 03:44:48.067759 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 03:44:48.067886 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260810 03:44:48.068183 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.058596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46212 (local address 127.21.175.125:41175)
0810 03:44:48.058962 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.058973 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.058990 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:44:48.059092 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.059097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.059151 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.059241 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.059699 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.060515 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.061793 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.062176 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.062281 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.064560 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:48.064565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:48.064572 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:48.064605 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:48.066475 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.066861 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.066864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.066866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.066966 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.067262 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.067265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.067267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.067467 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:48.067548 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:48.067816 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 03:44:48.067947 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":281,"thread_start_us":154,"threads_started":1}
I20260810 03:44:48.068907 14367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:44:48.069677 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.069407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:38751 (local address 127.0.0.1:55662)
0810 03:44:48.069534 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.069625 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 03:44:48.069938 14313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
I20260810 03:44:48.070879 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.070569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:38751 (local address 127.0.0.1:55668)
0810 03:44:48.070713 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.070809 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260810 03:44:48.071090 14367 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:38751: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.124.keytab.
Entry for principal kudu/127.21.175.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.124.keytab.
I20260810 03:44:48.072878 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.124:38751
--webserver_interface=127.21.175.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.124.keytab
--principal=kudu/127.21.175.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:37991,127.21.175.125:41175,127.21.175.124:38751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:44:48.097026 14284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } attempt: 1
I20260810 03:44:48.098084 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.097705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:38751 (local address 127.0.0.1:55674)
0810 03:44:48.097894 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.097972 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 03:44:48.098907 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.098654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:38751 (local address 127.0.0.1:55690)
0810 03:44:48.098780 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.098838 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 03:44:48.099069 14284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:38751: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
I20260810 03:44:48.120417 14367 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } attempt: 1
I20260810 03:44:48.121376 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.121065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:38751 (local address 127.0.0.1:55700)
0810 03:44:48.121220 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.121285 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 03:44:48.122191 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.121947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:38751 (local address 127.0.0.1:55706)
0810 03:44:48.122080 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.122125 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 03:44:48.122326 14367 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:38751: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
I20260810 03:44:48.174258 14284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } attempt: 2
I20260810 03:44:48.175220 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.174877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:38751 (local address 127.0.0.1:55720)
0810 03:44:48.175036 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.175106 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 03:44:48.175971 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.175760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:38751 (local address 127.0.0.1:55730)
0810 03:44:48.175868 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.175922 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 03:44:48.176120 14284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:38751: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
I20260810 03:44:48.190478 14367 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } attempt: 2
I20260810 03:44:48.191239 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.190955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:38751 (local address 127.0.0.1:55746)
0810 03:44:48.191100 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.191167 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 03:44:48.191947 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.191733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:38751 (local address 127.0.0.1:55750)
0810 03:44:48.191841 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.191891 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 03:44:48.192080 14367 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:38751: Network error: Client connection negotiation failed: client connection to 127.21.175.124:38751: connect: Connection refused (error 111)
W20260810 03:44:48.191846 14376 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:44:48.192257 14376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:48.192323 14376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:48.192402 14376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:48.196594 14376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:48.196684 14376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:48.196723 14376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:48.196761 14376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:48.196791 14376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:48.202337 14376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:37991,127.21.175.125:41175,127.21.175.124:38751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.124.keytab
--principal=kudu/127.21.175.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.124:38751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.175.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:48.203455 14376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:48.204504 14376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:48.213514 14384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:48.213464 14382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:48.213498 14381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:48.213557 14376 server_base.cc:1034] running on GCE node
I20260810 03:44:48.214380 14376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:48.215425 14376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:48.216615 14376 hybrid_clock.cc:648] HybridClock initialized: now 1786333488216598 us; error 30 us; skew 500 ppm
Aug 10 03:44:48 dist-test-slave-ss4n krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333488, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:48.219527 14376 init.cc:377] Logged in from keytab as kudu/127.21.175.124@KRBTEST.COM (short username kudu)
I20260810 03:44:48.220438 14376 webserver.cc:460] Webserver started at http://127.21.175.124:37257/ using document root <none> and password file <none>
I20260810 03:44:48.221002 14376 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:48.221098 14376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:48.221331 14376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:48.223564 14376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-2/data/instance:
uuid: "e5cd264fe7a6458989f192b1c86ecd88"
format_stamp: "Formatted at 2026-08-10 03:44:48 on dist-test-slave-ss4n"
server_key: "664ddbe879195745f01d1b8641898f82"
server_key_iv: "d3921b2cd9497824a5e42751543077c6"
server_key_version: "encryptionkey@0"
I20260810 03:44:48.224146 14376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-2/wal/instance:
uuid: "e5cd264fe7a6458989f192b1c86ecd88"
format_stamp: "Formatted at 2026-08-10 03:44:48 on dist-test-slave-ss4n"
server_key: "664ddbe879195745f01d1b8641898f82"
server_key_iv: "d3921b2cd9497824a5e42751543077c6"
server_key_version: "encryptionkey@0"
I20260810 03:44:48.227916 14376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260810 03:44:48.230723 14392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:48.232003 14376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 03:44:48.232143 14376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-2/wal
uuid: "e5cd264fe7a6458989f192b1c86ecd88"
format_stamp: "Formatted at 2026-08-10 03:44:48 on dist-test-slave-ss4n"
server_key: "664ddbe879195745f01d1b8641898f82"
server_key_iv: "d3921b2cd9497824a5e42751543077c6"
server_key_version: "encryptionkey@0"
I20260810 03:44:48.232256 14376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:48.251392 14376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:48.254985 14376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:48.255206 14376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:48.262449 14376 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.124:38751
I20260810 03:44:48.262804 14447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.124:38751 every 8 connection(s)
I20260810 03:44:48.263458 14376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-2/data/info.pb
I20260810 03:44:48.266489 14448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:48.270491 14448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:44:48.270884 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 14376
I20260810 03:44:48.271023 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333190020696-22205-0/minicluster-data/master-2/wal/instance
I20260810 03:44:48.271381 22205 external_mini_cluster.cc:1428] Setting key 4c67f1c253337d6fda3731ac6ba3a5a8
I20260810 03:44:48.275303 14284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } attempt: 3
Aug 10 03:44:48 dist-test-slave-ss4n krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for kudu/127.21.175.124@KRBTEST.COM
Aug 10 03:44:48 dist-test-slave-ss4n krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333488, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.124@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
Aug 10 03:44:48 dist-test-slave-ss4n krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.124@KRBTEST.COM
I20260810 03:44:48.283248 14367 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } attempt: 3
I20260810 03:44:48.286643 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.275706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:38751 (local address 127.0.0.1:55762)
0810 03:44:48.275878 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.275891 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:48.275960 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:48.276397 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:48.276399 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:48.276574 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:48.276668 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:48.276676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.277380 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.277384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:48.277975 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:48.277984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.278794 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.278798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:48.278894 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.279698 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:48.279715 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:48.281187 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:48.283828 (+ 2641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:48.283834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:48.283837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:48.284309 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:48.284770 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:48.284776 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:48.284778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:48.284884 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:48.286113 (+ 1229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:48.286116 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:48.286216 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 03:44:48.286312 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 03:44:48.286465 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260810 03:44:48.287966 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.272841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:37991 (local address 127.0.0.1:48920)
0810 03:44:48.273328 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.273378 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:48.274092 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:48.274649 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:48.274654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:48.275101 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:48.275493 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:48.275512 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.276638 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.276644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:48.278027 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:48.278070 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.278675 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.278680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:48.278830 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.279684 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:48.279708 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:48.282013 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:48.284512 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:48.284522 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:48.284531 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:48.284911 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:48.285254 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:48.285261 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:48.285266 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:48.285451 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:48.286107 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:48.286114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:48.286262 (+ 148us) client_negotiation.cc:769] Sending connection context
0810 03:44:48.286458 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 03:44:48.286833 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":144,"threads_started":1}
I20260810 03:44:48.288331 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.272839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.21.175.124:38751)
0810 03:44:48.273280 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.273294 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.273326 (+ 32us) server_negotiation.cc:407] Connection header received
0810 03:44:48.274061 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.274079 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.274295 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.274491 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.275402 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.276636 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.277981 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.278662 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.278825 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.282932 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:48.282937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:48.282941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:48.282971 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:48.286014 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.286510 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.286513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.286515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.286608 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.286922 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.286925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.286927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.287104 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:48.287944 (+ 840us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:48.288016 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 03:44:48.288132 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":312,"spinlock_wait_cycles":18560,"thread_start_us":172,"threads_started":1}
I20260810 03:44:48.287971 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.275775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55762 (local address 127.21.175.124:38751)
0810 03:44:48.276168 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.276177 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.276190 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:44:48.276224 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.276227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.276266 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.276361 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.276792 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.277269 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.278100 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.278663 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.279171 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.281341 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:48.281348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:48.281363 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:48.281400 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:48.283694 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.284457 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.284466 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.284475 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.284636 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.285356 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.285359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.285361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.285981 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:48.287251 (+ 1270us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:48.287363 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 03:44:48.287510 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":327,"thread_start_us":103,"threads_started":1}
I20260810 03:44:48.288882 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.272729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.21.175.126:37991)
0810 03:44:48.272847 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.272851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.273545 (+ 694us) server_negotiation.cc:407] Connection header received
0810 03:44:48.274390 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.274394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.274461 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.274590 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.275660 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.276518 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.278174 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.278494 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.278605 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.282207 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:48.282211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:48.282214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:48.282262 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:48.284384 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.285037 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.285042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.285045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.285133 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.285616 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.285620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.285622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.285936 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:48.286052 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:48.288582 (+ 2530us) server_negotiation.cc:299] Negotiation successful
0810 03:44:48.288709 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 03:44:48 dist-test-slave-ss4n krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.125@KRBTEST.COM for kudu/127.21.175.124@KRBTEST.COM
I20260810 03:44:48.290707 14448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:44:48.293675 14284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70: Bootstrap starting.
I20260810 03:44:48.295126 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.283743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:38751 (local address 127.0.0.1:55764)
0810 03:44:48.283976 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.283994 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:48.284092 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:48.284924 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:48.284928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:48.285123 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:48.285286 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:48.285312 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.286514 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.286517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:48.287316 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:48.287327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.287935 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.287939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:48.288022 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.288530 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:48.288545 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:48.290482 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:48.293456 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:48.293459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:48.293461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:48.293746 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:48.294095 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:48.294098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:48.294100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:48.294194 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:48.294620 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:48.294623 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:48.294715 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 03:44:48.294796 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 03:44:48.294925 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260810 03:44:48.295737 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.284327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.21.175.124:38751)
0810 03:44:48.284654 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.284661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.284674 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:44:48.284709 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.284713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.284760 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.285431 (+ 671us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.285566 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.286384 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.287462 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.287804 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.288237 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.291466 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:48.291470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:48.291472 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:48.291495 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:48.293327 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.293873 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.293876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.293877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.293970 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.294307 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.294310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.294312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.294506 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:48.295401 (+ 895us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:48.295475 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 03:44:48.295585 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":243,"thread_start_us":124,"threads_started":1}
I20260810 03:44:48.296764 14284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70: Neither blocks nor log segments found. Creating new log.
Aug 10 03:44:48 dist-test-slave-ss4n krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333488, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.124@KRBTEST.COM for kudu/127.21.175.125@KRBTEST.COM
I20260810 03:44:48.297791 14284 log.cc:824] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:44:48 dist-test-slave-ss4n krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:48.299247 14367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33: Bootstrap starting.
I20260810 03:44:48.302075 14367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:48.302692 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.291661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46216 (local address 127.21.175.125:41175)
0810 03:44:48.291793 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.291798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.291973 (+ 175us) server_negotiation.cc:407] Connection header received
0810 03:44:48.292145 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.292151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.292211 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.292266 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.292936 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.293872 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.294971 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.295338 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.295413 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.298521 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:48.298525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:48.298528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:48.298554 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:48.300592 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.301192 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.301196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.301197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.301287 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.301620 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.301623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.301625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.301921 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:48.302005 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:48.302420 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 03:44:48.302550 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:44:48.303113 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.291236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48932 (local address 127.21.175.126:37991)
0810 03:44:48.291359 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.291362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.291397 (+ 35us) server_negotiation.cc:407] Connection header received
0810 03:44:48.291436 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.291438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.291484 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.291558 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.293030 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.293824 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.294971 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.295331 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.295425 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.298546 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:48.298550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:48.298553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:48.298577 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:48.300469 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.300993 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.300997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.301001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.301093 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.301672 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.301676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.301677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.301904 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:48.302095 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:48.302832 (+ 737us) server_negotiation.cc:299] Negotiation successful
0810 03:44:48.302954 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:44:48.303216 14367 log.cc:824] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:48.303550 14284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70: No bootstrap required, opened a new log
I20260810 03:44:48.303802 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.291380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:41175 (local address 127.0.0.1:46216)
0810 03:44:48.291905 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.291920 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:48.292003 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:48.292437 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:48.292442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:48.292661 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:48.292792 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:48.292799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.294019 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.294022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:48.294826 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:48.294836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.295566 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.295570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:48.295665 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.296261 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:48.296273 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:48.298343 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:48.300683 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:48.300686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:48.300688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:48.300892 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:48.301382 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:48.301384 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:48.301386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:48.301459 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:48.302101 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:48.302104 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:48.302227 (+ 123us) client_negotiation.cc:769] Sending connection context
0810 03:44:48.303120 (+ 893us) client_negotiation.cc:240] Negotiation successful
0810 03:44:48.303658 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433}
I20260810 03:44:48.304724 14448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:44:48.305534 14367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33: No bootstrap required, opened a new log
I20260810 03:44:48.307868 14284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } }
I20260810 03:44:48.308117 14284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca202b1810041fabb3a83a50a253f70, State: Initialized, Role: FOLLOWER
I20260810 03:44:48.308470 14284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } }
I20260810 03:44:48.309051 14458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } } }
I20260810 03:44:48.309190 14458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:44:48.309356 14367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } }
I20260810 03:44:48.309583 14367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41311b868fe0469eac82046d4a9d1a33, State: Initialized, Role: FOLLOWER
I20260810 03:44:48.309908 14367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } }
I20260810 03:44:48.310102 14284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:48.310432 14459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } } }
I20260810 03:44:48.310650 14459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:44:48.311269 14367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33 [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 03:44:48 dist-test-slave-ss4n krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.125@KRBTEST.COM
Aug 10 03:44:48 dist-test-slave-ss4n krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333488, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.124@KRBTEST.COM for kudu/127.21.175.124@KRBTEST.COM
W20260810 03:44:48.315505 14471 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:44:48.315639 14471 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:44:48.318784 14292 master_runner.cc:186] Error getting master registration for 127.21.175.124:38751: OK, instance_id { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" instance_seqno: 1786333488260123 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:44:48.322476 14292 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:44:48.318966 14211 master_runner.cc:186] Error getting master registration for 127.21.175.124:38751: OK, instance_id { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" instance_seqno: 1786333488260123 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:44:48.322894 14211 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:44:48.323118 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.304640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46228 (local address 127.21.175.125:41175)
0810 03:44:48.305139 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.305143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.305156 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:44:48.305191 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.305195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.305237 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.305314 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.306243 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.306995 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.308123 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.308489 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.308605 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.316448 (+ 7843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:48.316452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:48.316455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:48.316481 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:48.319446 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.321023 (+ 1577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.321027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.321029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.321178 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.321535 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.321538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.321540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.321780 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:48.321990 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:48.322806 (+ 816us) server_negotiation.cc:299] Negotiation successful
0810 03:44:48.322942 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406}
W20260810 03:44:48.324512 14486 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:44:48.324604 14486 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:44:48.324976 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.305532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.21.175.124:38751)
0810 03:44:48.309692 (+ 4160us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.309696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.309712 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:44:48.309748 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.309751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.309792 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.309871 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.310634 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.311417 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.313097 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.313436 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.313550 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.316065 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:48.316070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:48.316073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:48.316113 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:48.320412 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.323725 (+ 3313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.323728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.323729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.323811 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.324070 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.324073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.324075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.324298 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:48.324372 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:48.324681 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 03:44:48.324816 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4030}
I20260810 03:44:48.325417 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.305173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:38751 (local address 127.0.0.1:55774)
0810 03:44:48.306317 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:48.306331 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:48.306398 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:48.310227 (+ 3829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:48.310230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:48.310396 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:48.310503 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:48.310510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.311541 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.311544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:48.312361 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:48.312372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.313704 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.313707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:48.313786 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.314315 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:48.314329 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:48.315926 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:48.323340 (+ 7414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:48.323343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:48.323346 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:48.323612 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:48.323898 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:48.323903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:48.323906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:48.323985 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:48.324427 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:48.324430 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:48.324505 (+ 75us) client_negotiation.cc:769] Sending connection context
0810 03:44:48.325123 (+ 618us) client_negotiation.cc:240] Negotiation successful
0810 03:44:48.325264 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084,"thread_start_us":119,"threads_started":1}
I20260810 03:44:48.327313 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:44:48.328778 14258 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9ca202b1810041fabb3a83a50a253f70"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48932
I20260810 03:44:48.329161 14258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 03:44:48.329244 14448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88: Bootstrap starting.
I20260810 03:44:48.329284 14258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:48.330683 14258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } }
I20260810 03:44:48.332290 14339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ca202b1810041fabb3a83a50a253f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "41311b868fe0469eac82046d4a9d1a33"
I20260810 03:44:48.332533 14339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:48.332975 14448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:48.333977 14448 log.cc:824] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:48.333990 14339 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca202b1810041fabb3a83a50a253f70 in term 1.
I20260810 03:44:48.334659 14234 leader_election.cc:302] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41311b868fe0469eac82046d4a9d1a33, 9ca202b1810041fabb3a83a50a253f70; no voters:
I20260810 03:44:48.334996 14458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:48.335170 14258 leader_election.cc:288] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [CANDIDATE]: Term 1 election: Requested vote from peers 41311b868fe0469eac82046d4a9d1a33 (127.21.175.125:41175), e5cd264fe7a6458989f192b1c86ecd88 (127.21.175.124:38751)
I20260810 03:44:48.335260 14458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [term 1 LEADER]: Becoming Leader. State: Replica: 9ca202b1810041fabb3a83a50a253f70, State: Running, Role: LEADER
I20260810 03:44:48.335642 14458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } }
I20260810 03:44:48.337677 14458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ca202b1810041fabb3a83a50a253f70. Latest consensus state: current_term: 1 leader_uuid: "9ca202b1810041fabb3a83a50a253f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } } }
I20260810 03:44:48.337795 14458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:48.351635 14495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:48.352763 14495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:48.357220 14448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88: No bootstrap required, opened a new log
I20260810 03:44:48.362340 14448 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } }
I20260810 03:44:48.362609 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.343038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.21.175.126:37991)
0810 03:44:48.343163 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.343167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.343730 (+ 563us) server_negotiation.cc:407] Connection header received
0810 03:44:48.349596 (+ 5866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.349600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.349665 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.349721 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.352480 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.353260 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.355065 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.355405 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.355480 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.358118 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:48.358123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:48.358125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:48.358156 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:48.360028 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.360808 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.360812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.360814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.360901 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.361353 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.361369 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.361370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.361620 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:48.361738 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:48.362308 (+ 570us) server_negotiation.cc:299] Negotiation successful
0810 03:44:48.362441 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:44:48.362824 14448 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5cd264fe7a6458989f192b1c86ecd88, State: Initialized, Role: FOLLOWER
I20260810 03:44:48.363270 14448 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } }
I20260810 03:44:48.364976 14448 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:48.365449 14421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ca202b1810041fabb3a83a50a253f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e5cd264fe7a6458989f192b1c86ecd88"
I20260810 03:44:48.365759 14421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:48.367240 14421 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca202b1810041fabb3a83a50a253f70 in term 1.
I20260810 03:44:48.368251 14504 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } } }
I20260810 03:44:48.369428 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.341988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55780 (local address 127.21.175.124:38751)
0810 03:44:48.342186 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.342191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.343362 (+ 1171us) server_negotiation.cc:407] Connection header received
0810 03:44:48.343535 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.343538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.343593 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.343653 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.344698 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.345524 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.346757 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.347119 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.347195 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.349147 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:48.349151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:48.349154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:48.349183 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:48.351004 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.364025 (+ 13021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.364029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.364031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.364152 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.364968 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.364973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.364975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.365228 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:48.365373 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:48.369144 (+ 3771us) server_negotiation.cc:299] Negotiation successful
0810 03:44:48.369275 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:44:48.373653 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:48.342554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46240 (local address 127.21.175.125:41175)
0810 03:44:48.342687 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.342693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.343363 (+ 670us) server_negotiation.cc:407] Connection header received
0810 03:44:48.343510 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.343513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.343571 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.343665 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.345149 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.345988 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.347738 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:48.348099 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:48.348177 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:48.369890 (+ 21713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:48.369894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:48.369897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:48.369932 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:48.371746 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.372283 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.372287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.372289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.372373 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:48.372668 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:48.372671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:48.372673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:48.372889 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:48.372987 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:48.373355 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 03:44:48.373474 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 03:44:48.374723 14504 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 03:44:48.377686 14516 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 03:44:48.378878 14339 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33 [term 1 FOLLOWER]: Refusing update from remote peer 9ca202b1810041fabb3a83a50a253f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:44:48.379638 14489 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:44:48.378839 14421 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88 [term 1 FOLLOWER]: Refusing update from remote peer 9ca202b1810041fabb3a83a50a253f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260810 03:44:48.381151 14516 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:44:48.382721 14489 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:44:48.383778 14459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ca202b1810041fabb3a83a50a253f70. Latest consensus state: current_term: 1 leader_uuid: "9ca202b1810041fabb3a83a50a253f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } } }
I20260810 03:44:48.383890 14459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:44:48.386593 14504 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ca202b1810041fabb3a83a50a253f70. Latest consensus state: current_term: 1 leader_uuid: "9ca202b1810041fabb3a83a50a253f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } } }
I20260810 03:44:48.386713 14504 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:44:48.389343 14489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ca202b1810041fabb3a83a50a253f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } } }
I20260810 03:44:48.389456 14489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:48.390728 14459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ca202b1810041fabb3a83a50a253f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } } }
I20260810 03:44:48.390836 14459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:44:48.391321 14501 mvcc.cc:205] Tried to move back new op lower bound from 7316821968391516160 to 7316821968226660352. Current Snapshot: MvccSnapshot[applied={T|T < 7316821968391516160}]
I20260810 03:44:48.391747 14518 mvcc.cc:205] Tried to move back new op lower bound from 7316821968391516160 to 7316821968226660352. Current Snapshot: MvccSnapshot[applied={T|T < 7316821968391516160}]
I20260810 03:44:48.395180 14495 catalog_manager.cc:1347] Generated new cluster ID: b083e754f20240d38bd2bcb5571613b0
I20260810 03:44:48.395241 14495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:48.414790 14458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ca202b1810041fabb3a83a50a253f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } } }
I20260810 03:44:48.414935 14458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:48.416276 14504 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ca202b1810041fabb3a83a50a253f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca202b1810041fabb3a83a50a253f70" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 37991 } } peers { permanent_uuid: "41311b868fe0469eac82046d4a9d1a33" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 41175 } } peers { permanent_uuid: "e5cd264fe7a6458989f192b1c86ecd88" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 38751 } } }
I20260810 03:44:48.416396 14504 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:44:48.417227 14520 mvcc.cc:205] Tried to move back new op lower bound from 7316821968391516160 to 7316821968226660352. Current Snapshot: MvccSnapshot[applied={T|T < 7316821968391516160}]
I20260810 03:44:48.426537 14495 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:48.427922 14495 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:48.442363 14495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ca202b1810041fabb3a83a50a253f70: Generated new TSK 0
I20260810 03:44:48.443414 14495 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:44:48.481377 14454 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:48.477813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.21.175.124:38751)
0810 03:44:48.477936 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.477940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.480351 (+ 2411us) server_negotiation.cc:407] Connection header received
0810 03:44:48.480535 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.480538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.480589 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.480641 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.481231 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55786: BlockingRecv error: recv got EOF from 127.0.0.1:55786 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 03:44:48.482223 14374 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:48.476800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.21.175.125:41175)
0810 03:44:48.476933 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.476939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.479248 (+ 2309us) server_negotiation.cc:407] Connection header received
0810 03:44:48.479477 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.479481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.479537 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.479603 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.482078 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46250: BlockingRecv error: recv got EOF from 127.0.0.1:46250 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 03:44:48.484272 14288 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:48.478540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.21.175.126:37991)
0810 03:44:48.478659 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.478663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.483025 (+ 4362us) server_negotiation.cc:407] Connection header received
0810 03:44:48.483466 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.483470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.483522 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.483574 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.484154 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48944: BlockingRecv error: recv got EOF from 127.0.0.1:48944 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 03:44:48.489300 14374 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:48.487592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46260 (local address 127.21.175.125:41175)
0810 03:44:48.487707 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.487711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.488262 (+ 551us) server_negotiation.cc:407] Connection header received
0810 03:44:48.488489 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.488495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.488580 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.488641 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.489192 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46260: BlockingRecv error: recv got EOF from 127.0.0.1:46260 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 03:44:48.494856 14374 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:48.493068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46274 (local address 127.21.175.125:41175)
0810 03:44:48.493182 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.493186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.493818 (+ 632us) server_negotiation.cc:407] Connection header received
0810 03:44:48.494059 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.494063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.494124 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.494183 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.494741 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46274: BlockingRecv error: recv got EOF from 127.0.0.1:46274 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 03:44:48.500842 14374 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:48.499123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46276 (local address 127.21.175.125:41175)
0810 03:44:48.499233 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.499237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.499818 (+ 581us) server_negotiation.cc:407] Connection header received
0810 03:44:48.500037 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.500041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.500111 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.500172 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.500727 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46276: BlockingRecv error: recv got EOF from 127.0.0.1:46276 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 03:44:48.509531 14374 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:48.507243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46288 (local address 127.21.175.125:41175)
0810 03:44:48.507415 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.507420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.508080 (+ 660us) server_negotiation.cc:407] Connection header received
0810 03:44:48.508308 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.508312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.508391 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.508450 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.509290 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46288: BlockingRecv error: recv got EOF from 127.0.0.1:46288 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260810 03:44:48.522468 14374 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:48.520328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46290 (local address 127.21.175.125:41175)
0810 03:44:48.520505 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.520510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.521212 (+ 702us) server_negotiation.cc:407] Connection header received
0810 03:44:48.521467 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.521471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.521574 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.521638 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.522318 (+ 680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46290: BlockingRecv error: recv got EOF from 127.0.0.1:46290 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260810 03:44:48.542574 14374 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:48.541154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46292 (local address 127.21.175.125:41175)
0810 03:44:48.541289 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.541294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.541609 (+ 315us) server_negotiation.cc:407] Connection header received
0810 03:44:48.541788 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.541796 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.541847 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.541944 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.542446 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46292: BlockingRecv error: recv got EOF from 127.0.0.1:46292 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 03:44:48.577960 14374 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:48.576697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.21.175.125:41175)
0810 03:44:48.576846 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.576851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.576974 (+ 123us) server_negotiation.cc:407] Connection header received
0810 03:44:48.577100 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.577104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.577166 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.577274 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.577825 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46304: BlockingRecv error: recv got EOF from 127.0.0.1:46304 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 03:44:48.645337 14374 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:48.644182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46320 (local address 127.21.175.125:41175)
0810 03:44:48.644357 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.644364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.644462 (+ 98us) server_negotiation.cc:407] Connection header received
0810 03:44:48.644605 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.644608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.644664 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.644756 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.645230 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46320: BlockingRecv error: recv got EOF from 127.0.0.1:46320 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 03:44:48.777192 14374 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:48.775852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46328 (local address 127.21.175.125:41175)
0810 03:44:48.775989 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:48.775993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:48.776234 (+ 241us) server_negotiation.cc:407] Connection header received
0810 03:44:48.776371 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:48.776374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:48.776476 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:48.776573 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:48.777069 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46328: BlockingRecv error: recv got EOF from 127.0.0.1:46328 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 03:44:49.036718 14374 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:49.035553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.21.175.125:41175)
0810 03:44:49.035671 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:49.035676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:49.035802 (+ 126us) server_negotiation.cc:407] Connection header received
0810 03:44:49.035909 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:49.035913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:49.035967 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:49.036072 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:49.036604 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46336: BlockingRecv error: recv got EOF from 127.0.0.1:46336 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 03:44:49.326169 14486 catalog_manager.cc:1259] Loaded cluster ID: b083e754f20240d38bd2bcb5571613b0
I20260810 03:44:49.326284 14486 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33: loading cluster ID for follower catalog manager: success
I20260810 03:44:49.327965 14486 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33: acquiring CA information for follower catalog manager: success
I20260810 03:44:49.329190 14486 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 41311b868fe0469eac82046d4a9d1a33: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:44:49.382666 14516 catalog_manager.cc:1259] Loaded cluster ID: b083e754f20240d38bd2bcb5571613b0
I20260810 03:44:49.382752 14516 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88: loading cluster ID for follower catalog manager: success
I20260810 03:44:49.384505 14516 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88: acquiring CA information for follower catalog manager: success
I20260810 03:44:49.385746 14516 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5cd264fe7a6458989f192b1c86ecd88: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:44:49.555855 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:49.551508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46350 (local address 127.21.175.125:41175)
0810 03:44:49.551778 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:49.551782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:49.551880 (+ 98us) server_negotiation.cc:407] Connection header received
0810 03:44:49.552026 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:49.552029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:49.552164 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:49.552276 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:44:49.552608 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:49.553184 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:49.554233 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:49.554531 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:49.554636 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:49.554986 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:44:49.555446 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:44:49.555553 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:49.555668 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 03:44:49.555726 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":111,"threads_started":1}
I20260810 03:44:49.563535 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:49.559479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55790 (local address 127.21.175.124:38751)
0810 03:44:49.559795 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:49.559801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:49.559816 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:44:49.560023 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:49.560029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:49.560158 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:49.560290 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:44:49.560519 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:49.561061 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:49.562000 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:49.562243 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:49.562355 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:49.562674 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:44:49.563128 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:44:49.563272 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:49.563313 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 03:44:49.563366 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":69,"threads_started":1}
I20260810 03:44:49.571182 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:49.567138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55792 (local address 127.21.175.124:38751)
0810 03:44:49.567292 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:49.567309 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:49.567628 (+ 319us) server_negotiation.cc:407] Connection header received
0810 03:44:49.567741 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:49.567745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:49.567797 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:49.567875 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:44:49.568175 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:49.568845 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:49.569940 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:49.570283 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:49.570410 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:49.570647 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:44:49.570727 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:44:49.570803 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:49.570931 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 03:44:49.570985 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:44:49.571904 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:49.567279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.21.175.125:41175)
0810 03:44:49.567410 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:49.567414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:49.567920 (+ 506us) server_negotiation.cc:407] Connection header received
0810 03:44:49.568128 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:49.568134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:49.568197 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:49.568247 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:44:49.568620 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:49.569373 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:49.570473 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:49.570790 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:49.570867 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:49.571414 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:44:49.571495 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:44:49.571538 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:49.571707 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 03:44:49.571762 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:44:49.572269 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:49.567207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48960 (local address 127.21.175.126:37991)
0810 03:44:49.567445 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:49.567448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:49.567586 (+ 138us) server_negotiation.cc:407] Connection header received
0810 03:44:49.567763 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:49.567765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:49.567930 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:49.568076 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:44:49.568351 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:49.569187 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:49.570231 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:49.570528 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:49.570639 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:49.571228 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:44:49.571609 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:44:49.571697 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:49.572048 (+ 351us) server_negotiation.cc:299] Negotiation successful
0810 03:44:49.572113 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260810 03:44:49.581399 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 14211
I20260810 03:44:49.591092 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 14292
I20260810 03:44:49.599862 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 14376
2026-08-10T03:44:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3244 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:44:49 dist-test-slave-ss4n krb5kdc[14612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:44:49 dist-test-slave-ss4n krb5kdc[14612](info): set up 2 sockets
Aug 10 03:44:49 dist-test-slave-ss4n krb5kdc[14612](info): commencing operation
krb5kdc: starting...
W20260810 03:44:50.746543 22205 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/test-user.keytab.
Aug 10 03:44:50 dist-test-slave-ss4n krb5kdc[14612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:44:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:44:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
Entry for principal kudu/127.21.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab.
I20260810 03:44:50.884812 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:44865
--webserver_interface=127.21.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:35833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:44865,127.21.175.125:38491,127.21.175.124:42341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:50.999068 14626 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:44:50.999361 14626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:50.999383 14626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:50.999470 14626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:51.003610 14626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:51.003676 14626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:51.003696 14626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:51.003713 14626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:51.003729 14626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:51.009079 14626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:35833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:44865,127.21.175.125:38491,127.21.175.124:42341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.126.keytab
--principal=kudu/127.21.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.126:44865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:51.010282 14626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:51.011363 14626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:51.020390 14631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:51.020403 14632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:51.020642 14626 server_base.cc:1034] running on GCE node
W20260810 03:44:51.020390 14634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:51.021412 14626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:51.022650 14626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:51.023849 14626 hybrid_clock.cc:648] HybridClock initialized: now 1786333491023835 us; error 27 us; skew 500 ppm
Aug 10 03:44:51 dist-test-slave-ss4n krb5kdc[14612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:51.026736 14626 init.cc:377] Logged in from keytab as kudu/127.21.175.126@KRBTEST.COM (short username kudu)
I20260810 03:44:51.027638 14626 webserver.cc:460] Webserver started at http://127.21.175.126:37763/ using document root <none> and password file <none>
I20260810 03:44:51.028205 14626 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:51.028297 14626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:51.028537 14626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:51.030766 14626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-0/data/instance:
uuid: "4fa1a3132d7a42a6ac4de30c10769db4"
format_stamp: "Formatted at 2026-08-10 03:44:51 on dist-test-slave-ss4n"
server_key: "346461430ae23d4d4044a1334b2281a6"
server_key_iv: "4af4b6aeb35c49bd709b8d6b63386aab"
server_key_version: "encryptionkey@0"
I20260810 03:44:51.031387 14626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-0/wal/instance:
uuid: "4fa1a3132d7a42a6ac4de30c10769db4"
format_stamp: "Formatted at 2026-08-10 03:44:51 on dist-test-slave-ss4n"
server_key: "346461430ae23d4d4044a1334b2281a6"
server_key_iv: "4af4b6aeb35c49bd709b8d6b63386aab"
server_key_version: "encryptionkey@0"
I20260810 03:44:51.035225 14626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:44:51.037887 14641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:51.039407 14626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:44:51.039628 14626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-0/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-0/wal
uuid: "4fa1a3132d7a42a6ac4de30c10769db4"
format_stamp: "Formatted at 2026-08-10 03:44:51 on dist-test-slave-ss4n"
server_key: "346461430ae23d4d4044a1334b2281a6"
server_key_iv: "4af4b6aeb35c49bd709b8d6b63386aab"
server_key_version: "encryptionkey@0"
I20260810 03:44:51.039764 14626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:51.067830 14626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:51.071305 14626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:51.071535 14626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:51.078874 14626 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.126:44865
I20260810 03:44:51.079217 14701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.126:44865 every 8 connection(s)
I20260810 03:44:51.079896 14626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-0/data/info.pb
I20260810 03:44:51.082597 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 14626
I20260810 03:44:51.082733 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-0/wal/instance
I20260810 03:44:51.083017 22205 external_mini_cluster.cc:1428] Setting key 1e4e4b6920c817676a6e8b196108ab8c
I20260810 03:44:51.083218 14702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:51.088167 14702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.175.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.125@KRBTEST.COM" created.
Aug 10 03:44:51 dist-test-slave-ss4n krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:51.105893 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.090710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:44865 (local address 127.0.0.1:37090)
0810 03:44:51.091236 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.091294 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:51.092283 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:51.093119 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:51.093127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:51.093604 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:51.093888 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.093903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.095250 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.095255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.096637 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.096654 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.097351 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.097355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.097447 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.098149 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:51.098166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:51.099935 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:51.102448 (+ 2513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.102459 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.102472 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.102812 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.103171 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.103177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.103180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.103346 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.104122 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:51.104131 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:51.104324 (+ 193us) client_negotiation.cc:769] Sending connection context
0810 03:44:51.104547 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 03:44:51.104966 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":102,"thread_start_us":168,"threads_started":1}
I20260810 03:44:51.105893 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.090711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37090 (local address 127.21.175.126:44865)
0810 03:44:51.091336 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.091348 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.091398 (+ 50us) server_negotiation.cc:407] Connection header received
0810 03:44:51.092596 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.092600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.092795 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.093004 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.094046 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.095075 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.096789 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.097183 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.097363 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.100107 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:51.100118 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:51.100135 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:51.100170 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:51.102335 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.102943 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.102949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.102951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.103070 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.103533 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.103536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.103537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.103970 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:51.104092 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:51.104626 (+ 534us) server_negotiation.cc:299] Negotiation successful
0810 03:44:51.104852 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":495,"spinlock_wait_cycles":131840,"thread_start_us":164,"threads_started":1}
I20260810 03:44:51.108464 14702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:44:51.109221 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.108944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:38491 (local address 127.0.0.1:49958)
0810 03:44:51.109068 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.109162 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 03:44:51.109460 14645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.175.125:38491: connect: Connection refused (error 111)
I20260810 03:44:51.110461 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.110216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:38491 (local address 127.0.0.1:49972)
0810 03:44:51.110365 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.110411 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260810 03:44:51.110849 14702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.125:38491: Network error: Client connection negotiation failed: client connection to 127.21.175.125:38491: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.125.keytab.
Entry for principal kudu/127.21.175.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.125.keytab.
I20260810 03:44:51.114590 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.125:38491
--webserver_interface=127.21.175.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.125.keytab
--principal=kudu/127.21.175.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:35833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:44865,127.21.175.125:38491,127.21.175.124:42341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:44:51.160204 14702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } attempt: 1
I20260810 03:44:51.161175 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.160823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:38491 (local address 127.0.0.1:49984)
0810 03:44:51.161006 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.161084 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 03:44:51.161970 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.161734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:38491 (local address 127.0.0.1:49986)
0810 03:44:51.161841 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.161897 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 03:44:51.162127 14702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.125:38491: Network error: Client connection negotiation failed: client connection to 127.21.175.125:38491: connect: Connection refused (error 111)
I20260810 03:44:51.230327 14702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } attempt: 2
I20260810 03:44:51.231346 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.231007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:38491 (local address 127.0.0.1:49998)
0810 03:44:51.231184 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.231257 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 03:44:51.232115 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.231890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:38491 (local address 127.0.0.1:50002)
0810 03:44:51.232006 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.232058 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.125:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 03:44:51.232259 14702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.125:38491: Network error: Client connection negotiation failed: client connection to 127.21.175.125:38491: connect: Connection refused (error 111)
W20260810 03:44:51.232632 14709 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:44:51.232939 14709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:51.233000 14709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:51.233072 14709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:44:51.237110 14709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:51.237195 14709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:51.237242 14709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:51.237299 14709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:51.237336 14709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:51.242895 14709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:35833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:44865,127.21.175.125:38491,127.21.175.124:42341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.125.keytab
--principal=kudu/127.21.175.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.125:38491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.175.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:51.244020 14709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:51.245085 14709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:51.253764 14717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:51.253810 14714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:51.253835 14709 server_base.cc:1034] running on GCE node
W20260810 03:44:51.253782 14715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:51.254588 14709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:51.255734 14709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:51.256897 14709 hybrid_clock.cc:648] HybridClock initialized: now 1786333491256884 us; error 25 us; skew 500 ppm
Aug 10 03:44:51 dist-test-slave-ss4n krb5kdc[14612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:51.259918 14709 init.cc:377] Logged in from keytab as kudu/127.21.175.125@KRBTEST.COM (short username kudu)
I20260810 03:44:51.260823 14709 webserver.cc:460] Webserver started at http://127.21.175.125:45851/ using document root <none> and password file <none>
I20260810 03:44:51.261399 14709 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:51.261513 14709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:51.261765 14709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:51.263885 14709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-1/data/instance:
uuid: "4cd8f428b1e94d91a36dccd9aea11df6"
format_stamp: "Formatted at 2026-08-10 03:44:51 on dist-test-slave-ss4n"
server_key: "dd3b87f30f89c65e896865cc767d69fc"
server_key_iv: "29f5cc479f425900c4490351cd8962c1"
server_key_version: "encryptionkey@0"
I20260810 03:44:51.264464 14709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-1/wal/instance:
uuid: "4cd8f428b1e94d91a36dccd9aea11df6"
format_stamp: "Formatted at 2026-08-10 03:44:51 on dist-test-slave-ss4n"
server_key: "dd3b87f30f89c65e896865cc767d69fc"
server_key_iv: "29f5cc479f425900c4490351cd8962c1"
server_key_version: "encryptionkey@0"
I20260810 03:44:51.268522 14709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 03:44:51.271598 14724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:51.272874 14709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:44:51.272979 14709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-1/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-1/wal
uuid: "4cd8f428b1e94d91a36dccd9aea11df6"
format_stamp: "Formatted at 2026-08-10 03:44:51 on dist-test-slave-ss4n"
server_key: "dd3b87f30f89c65e896865cc767d69fc"
server_key_iv: "29f5cc479f425900c4490351cd8962c1"
server_key_version: "encryptionkey@0"
I20260810 03:44:51.273118 14709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:51.304471 14709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:51.308097 14709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:51.308319 14709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:51.315807 14709 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.125:38491
I20260810 03:44:51.316124 14780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.125:38491 every 8 connection(s)
I20260810 03:44:51.316906 14709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-1/data/info.pb
I20260810 03:44:51.319841 14781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:51.322485 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 14709
I20260810 03:44:51.322655 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-1/wal/instance
I20260810 03:44:51.322994 22205 external_mini_cluster.cc:1428] Setting key f711add925a3ec74a3424fe65c5743d6
I20260810 03:44:51.323447 14702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } attempt: 3
I20260810 03:44:51.324129 14781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.175.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.175.124@KRBTEST.COM" created.
Aug 10 03:44:51 dist-test-slave-ss4n krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for kudu/127.21.175.125@KRBTEST.COM
Aug 10 03:44:51 dist-test-slave-ss4n krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.125@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:51.342492 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.324047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:38491 (local address 127.0.0.1:50012)
0810 03:44:51.324294 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.324312 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:51.324409 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:51.327464 (+ 3055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:51.327470 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:51.327706 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:51.327876 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.327887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.329594 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.329601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.330541 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.330553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.331629 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.331633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.331782 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.332431 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:51.332443 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:51.337646 (+ 5203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:51.340349 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.340352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.340354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.340602 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.340993 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.340996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.340998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.341076 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.342005 (+ 929us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:51.342010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:51.342129 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 03:44:51.342248 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 03:44:51.342364 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260810 03:44:51.342492 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.326423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37100 (local address 127.21.175.126:44865)
0810 03:44:51.326558 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.326562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.326922 (+ 360us) server_negotiation.cc:407] Connection header received
0810 03:44:51.327385 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.327393 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.327461 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.327555 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.328645 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.329519 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.331212 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.332167 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.332300 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.338398 (+ 6098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:51.338405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:51.338412 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:51.338446 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:51.340118 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.340753 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.340756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.340757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.340838 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.341226 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.341229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.341231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.341428 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:51.341592 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:51.342143 (+ 551us) server_negotiation.cc:299] Negotiation successful
0810 03:44:51.342307 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 03:44:51.343989 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.324945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.21.175.125:38491)
0810 03:44:51.325643 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.325660 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.325699 (+ 39us) server_negotiation.cc:407] Connection header received
0810 03:44:51.326770 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.326795 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.327172 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.327496 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.328069 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.329411 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.330689 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.331473 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.331673 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.337857 (+ 6184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:51.337868 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:51.337888 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:51.337926 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:51.340237 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.340732 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.340741 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.340746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.340895 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.341180 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.341185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.341187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.341871 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:51.342019 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:51.342403 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 03:44:51.342784 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":476,"spinlock_wait_cycles":51712,"thread_start_us":178,"threads_started":1}
I20260810 03:44:51.344007 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.326359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:44865 (local address 127.0.0.1:37100)
0810 03:44:51.326765 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.326826 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:51.327157 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:51.327724 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:51.327730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:51.328230 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:51.328455 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.328469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.329649 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.329652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.331072 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.331083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.332943 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.332953 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.333057 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.333651 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:51.333669 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:51.338230 (+ 4561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:51.340252 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.340263 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.340276 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.340629 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.340936 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.340939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.340941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.341109 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.341584 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:51.341592 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:51.341847 (+ 255us) client_negotiation.cc:769] Sending connection context
0810 03:44:51.342529 (+ 682us) client_negotiation.cc:240] Negotiation successful
0810 03:44:51.343207 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":119,"spinlock_wait_cycles":3968,"thread_start_us":102,"threads_started":1}
I20260810 03:44:51.345957 14781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:44:51.346901 14702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:44:51.347702 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.347387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:42341 (local address 127.0.0.1:49388)
0810 03:44:51.347553 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.347616 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260810 03:44:51.348444 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.348182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:42341 (local address 127.0.0.1:49396)
0810 03:44:51.348318 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.348366 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260810 03:44:51.348650 14702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:42341: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
Aug 10 03:44:51 dist-test-slave-ss4n krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.125@KRBTEST.COM for kudu/127.21.175.125@KRBTEST.COM
I20260810 03:44:51.355525 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.346555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:38491 (local address 127.0.0.1:50028)
0810 03:44:51.346695 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.346707 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:51.346810 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:51.347223 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:51.347227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:51.347415 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:51.347531 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.347538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.348463 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.348466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.349142 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.349152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.349790 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.349795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.349879 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.350408 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:51.350419 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:51.351759 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:51.353892 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.353894 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.353896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.354086 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.354414 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.354416 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.354417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.354499 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.355041 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:51.355043 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:51.355131 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 03:44:51.355217 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 03:44:51.355330 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 03:44:51.355556 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.346566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.21.175.125:38491)
0810 03:44:51.346954 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.346957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.346968 (+ 11us) server_negotiation.cc:407] Connection header received
0810 03:44:51.346998 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.347002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.347044 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.347115 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.347682 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.348351 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.349256 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.349651 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.349777 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.351874 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:51.351877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:51.351879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:51.351901 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:51.353788 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.354193 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.354195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.354197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.354305 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.354657 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.354660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.354672 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.354939 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:51.355040 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:51.355280 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 03:44:51.355439 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317}
I20260810 03:44:51.356722 14781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.175.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.124.keytab.
Entry for principal kudu/127.21.175.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.124.keytab.
I20260810 03:44:51.357707 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.357331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:42341 (local address 127.0.0.1:49404)
0810 03:44:51.357526 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.357642 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 03:44:51.358031 22205 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
/tmp/dist-test-taskMmCxq6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.124:42341
--webserver_interface=127.21.175.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.124.keytab
--principal=kudu/127.21.175.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.175.84:35833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.175.126:44865,127.21.175.125:38491,127.21.175.124:42341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:44:51.358501 14727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
I20260810 03:44:51.359524 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.359306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:42341 (local address 127.0.0.1:49410)
0810 03:44:51.359422 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.359469 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 03:44:51.359737 14781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:42341: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
I20260810 03:44:51.379869 14702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } attempt: 1
I20260810 03:44:51.380851 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.380479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:42341 (local address 127.0.0.1:49426)
0810 03:44:51.380673 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.380754 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 03:44:51.381796 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.381456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:42341 (local address 127.0.0.1:49438)
0810 03:44:51.381673 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.381729 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260810 03:44:51.381964 14702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:42341: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
I20260810 03:44:51.409068 14781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } attempt: 1
I20260810 03:44:51.410097 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.409750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:42341 (local address 127.0.0.1:49448)
0810 03:44:51.409921 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.409993 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 03:44:51.410950 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.410727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:42341 (local address 127.0.0.1:49462)
0810 03:44:51.410839 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.410891 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 03:44:51.411152 14781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:42341: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
I20260810 03:44:51.457206 14702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } attempt: 2
I20260810 03:44:51.458209 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.457832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:42341 (local address 127.0.0.1:49476)
0810 03:44:51.458014 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.458104 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 03:44:51.459029 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.458782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:42341 (local address 127.0.0.1:49482)
0810 03:44:51.458903 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.458961 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 03:44:51.459235 14702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:42341: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
W20260810 03:44:51.476044 14789 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:44:51.476399 14789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:44:51.476428 14789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:44:51.476507 14789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 03:44:51.479343 14781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } attempt: 2
I20260810 03:44:51.480191 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.479896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:42341 (local address 127.0.0.1:49494)
0810 03:44:51.480049 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.480115 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 03:44:51.480898 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.480709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:42341 (local address 127.0.0.1:49498)
0810 03:44:51.480806 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.480849 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 03:44:51.481069 14781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.175.124:42341: Network error: Client connection negotiation failed: client connection to 127.21.175.124:42341: connect: Connection refused (error 111)
W20260810 03:44:51.481325 14789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:44:51.481402 14789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:44:51.481447 14789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:44:51.481513 14789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:44:51.481550 14789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:44:51.487028 14789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.175.84:35833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.175.126:44865,127.21.175.125:38491,127.21.175.124:42341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/krb5kdc/kudu_127.21.175.124.keytab
--principal=kudu/127.21.175.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.175.124:42341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.175.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:44:51.488193 14789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:44:51.489213 14789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:44:51.498152 14798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:51.498176 14795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:44:51.498287 14796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:44:51.498446 14789 server_base.cc:1034] running on GCE node
I20260810 03:44:51.499315 14789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:44:51.500618 14789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:44:51.501837 14789 hybrid_clock.cc:648] HybridClock initialized: now 1786333491501825 us; error 30 us; skew 500 ppm
Aug 10 03:44:51 dist-test-slave-ss4n krb5kdc[14612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:44:51.505182 14789 init.cc:377] Logged in from keytab as kudu/127.21.175.124@KRBTEST.COM (short username kudu)
I20260810 03:44:51.506392 14789 webserver.cc:460] Webserver started at http://127.21.175.124:41967/ using document root <none> and password file <none>
I20260810 03:44:51.507076 14789 fs_manager.cc:359] Metadata directory not provided
I20260810 03:44:51.507181 14789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:44:51.507445 14789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:44:51.509831 14789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-2/data/instance:
uuid: "9ca626539aba4d098275bde40cbc9f6e"
format_stamp: "Formatted at 2026-08-10 03:44:51 on dist-test-slave-ss4n"
server_key: "7188d42a7a80d63456ab0a49af0a3c42"
server_key_iv: "3a2025347803cae2518560b566199439"
server_key_version: "encryptionkey@0"
I20260810 03:44:51.510509 14789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-2/wal/instance:
uuid: "9ca626539aba4d098275bde40cbc9f6e"
format_stamp: "Formatted at 2026-08-10 03:44:51 on dist-test-slave-ss4n"
server_key: "7188d42a7a80d63456ab0a49af0a3c42"
server_key_iv: "3a2025347803cae2518560b566199439"
server_key_version: "encryptionkey@0"
I20260810 03:44:51.514808 14789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:44:51.518146 14805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:44:51.519466 14789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:44:51.519654 14789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-2/data,/tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-2/wal
uuid: "9ca626539aba4d098275bde40cbc9f6e"
format_stamp: "Formatted at 2026-08-10 03:44:51 on dist-test-slave-ss4n"
server_key: "7188d42a7a80d63456ab0a49af0a3c42"
server_key_iv: "3a2025347803cae2518560b566199439"
server_key_version: "encryptionkey@0"
I20260810 03:44:51.519788 14789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:44:51.531951 14789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:44:51.535089 14789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:44:51.535311 14789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:44:51.542689 14789 rpc_server.cc:307] RPC server started. Bound to: 127.21.175.124:42341
I20260810 03:44:51.543008 14861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.175.124:42341 every 8 connection(s)
I20260810 03:44:51.543802 14789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-2/data/info.pb
I20260810 03:44:51.545993 22205 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu as pid 14789
I20260810 03:44:51.546140 22205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMmCxq6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333190020696-22205-0/minicluster-data/master-2/wal/instance
I20260810 03:44:51.546424 22205 external_mini_cluster.cc:1428] Setting key 5ba2fe0050aafc1e7c81206385201668
I20260810 03:44:51.546988 14862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:44:51.551767 14862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:44:51 dist-test-slave-ss4n krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.124@KRBTEST.COM
I20260810 03:44:51.558915 14702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } attempt: 3
Aug 10 03:44:51 dist-test-slave-ss4n krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.124@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:51.566295 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.547860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49506 (local address 127.21.175.124:42341)
0810 03:44:51.548269 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.548288 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.548335 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:44:51.549004 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.549022 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.549369 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.549724 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.550660 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.551667 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.552828 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.553353 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.553562 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.556314 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:51.556320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:51.556332 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:51.556362 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:51.558680 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.561907 (+ 3227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.561913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.561920 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.562056 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.562453 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.562458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.562461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.564076 (+ 1615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:51.564292 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:51.564646 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 03:44:51.564932 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":142,"threads_started":1}
Aug 10 03:44:51 dist-test-slave-ss4n krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.126@KRBTEST.COM for kudu/127.21.175.124@KRBTEST.COM
I20260810 03:44:51.572491 14781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } attempt: 3
I20260810 03:44:51.572788 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.553763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.21.175.126:44865)
0810 03:44:51.553914 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.553919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.554182 (+ 263us) server_negotiation.cc:407] Connection header received
0810 03:44:51.554339 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.554343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.554405 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.554503 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.555165 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.555700 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.556811 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.557036 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.557149 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.562009 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:51.562015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:51.562020 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:51.562066 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:51.564182 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.570070 (+ 5888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.570074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.570076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.570168 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.570479 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.570483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.570484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.571726 (+ 1242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:51.571825 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:51.572237 (+ 412us) server_negotiation.cc:299] Negotiation successful
0810 03:44:51.572366 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:44:51.572947 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.560075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:42341 (local address 127.0.0.1:49518)
0810 03:44:51.560373 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.560390 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:51.560475 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:51.562228 (+ 1753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:51.562232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:51.562733 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:51.562874 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.562885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.564097 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.564102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.564842 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.564853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.565541 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.565545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.565625 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.566126 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:51.566145 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:51.567451 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:51.569783 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.569789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.569794 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.570080 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.570447 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.570452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.570457 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.571483 (+ 1026us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.572334 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:51.572339 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:51.572455 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 03:44:51.572547 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 03:44:51.572792 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260810 03:44:51.573439 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.553660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.126:44865 (local address 127.0.0.1:37108)
0810 03:44:51.554018 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.554085 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:51.554222 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:51.554554 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:51.554559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:51.554899 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:51.555020 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.555025 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.555828 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.555832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.556686 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.556696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.557151 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.557155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.557276 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.557759 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:51.557778 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:51.561825 (+ 4047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:51.569610 (+ 7785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.569618 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.569622 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.569931 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.570281 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.570284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.570285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.570373 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.571871 (+ 1498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:51.571875 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:51.571985 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 03:44:51.573039 (+ 1054us) client_negotiation.cc:240] Negotiation successful
0810 03:44:51.573251 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":122,"threads_started":1}
I20260810 03:44:51.574622 14862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:44:51.578502 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.561614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.21.175.124:42341)
0810 03:44:51.561973 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.561986 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.561999 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:44:51.562028 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.562031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.562068 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.562999 (+ 931us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.563099 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.563964 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.564971 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.565371 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.565528 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.567604 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:51.567610 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:51.567619 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:51.567656 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:51.569654 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.570215 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.570221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.570225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.570337 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.571650 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.571654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.571658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.572198 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:51.578004 (+ 5806us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:51.578094 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 03:44:51.578237 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":145,"threads_started":1}
Aug 10 03:44:51 dist-test-slave-ss4n krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.125@KRBTEST.COM for kudu/127.21.175.124@KRBTEST.COM
Aug 10 03:44:51 dist-test-slave-ss4n krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.126@KRBTEST.COM
I20260810 03:44:51.582377 14702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4: Bootstrap starting.
I20260810 03:44:51.584096 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.572920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:42341 (local address 127.0.0.1:49530)
0810 03:44:51.573194 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.573210 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:51.573296 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:51.574018 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:51.574024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:51.574223 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:51.574326 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.574334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.575405 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.575410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.576240 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.576251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.576905 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.576908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.576984 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.577585 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:51.577599 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:51.579813 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:51.582207 (+ 2394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.582210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.582212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.582741 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.583075 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.583078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.583080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.583169 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.583590 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:51.583593 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:51.583692 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 03:44:51.583787 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 03:44:51.583931 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260810 03:44:51.584597 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.573441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49530 (local address 127.21.175.124:42341)
0810 03:44:51.573789 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.573794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.573806 (+ 12us) server_negotiation.cc:407] Connection header received
0810 03:44:51.573836 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.573838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.573879 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.574438 (+ 559us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.574523 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.575276 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.576398 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.576784 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.579954 (+ 3170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.580071 (+ 117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:51.580074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:51.580077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:51.580093 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:51.581808 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.582869 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.582872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.582874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.582955 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.583280 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.583283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.583284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.583474 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:51.584288 (+ 814us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:51.584364 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 03:44:51.584468 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274}
I20260810 03:44:51.585812 14702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:51.586386 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.572519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.21.175.126:44865)
0810 03:44:51.573010 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.573014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.573422 (+ 408us) server_negotiation.cc:407] Connection header received
0810 03:44:51.573460 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.573465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.573533 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.573626 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.574614 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.575471 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.576853 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.577256 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.577345 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.581523 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:51.581528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:51.581533 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:51.581578 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:51.583379 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.583956 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.583961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.583965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.584059 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.585013 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.585019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.585024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.585284 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:51.585451 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:51.586004 (+ 553us) server_negotiation.cc:299] Negotiation successful
0810 03:44:51.586146 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":72,"threads_started":1}
I20260810 03:44:51.586797 14702 log.cc:824] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:51.587613 14781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6: Bootstrap starting.
Aug 10 03:44:51 dist-test-slave-ss4n krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.124@KRBTEST.COM for kudu/127.21.175.125@KRBTEST.COM
I20260810 03:44:51.590451 14781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:51.591886 14781 log.cc:824] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:51.594377 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.575031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.125:38491 (local address 127.0.0.1:50032)
0810 03:44:51.575581 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.575595 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:51.575699 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:51.575987 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:51.575989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:51.576151 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:51.576258 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.576264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.577231 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.577235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.578090 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.578103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.578860 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.578865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.578957 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.587424 (+ 8467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:51.587438 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:51.589994 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:51.592276 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.592281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.592286 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.592584 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.592948 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.592953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.592956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.593058 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.593695 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:51.593700 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:51.593830 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 03:44:51.593952 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 03:44:51.594119 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":104,"threads_started":1}
I20260810 03:44:51.594617 14702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4: No bootstrap required, opened a new log
Aug 10 03:44:51 dist-test-slave-ss4n krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.175.125@KRBTEST.COM
I20260810 03:44:51.595165 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.575344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.21.175.125:38491)
0810 03:44:51.575462 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.575465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.575642 (+ 177us) server_negotiation.cc:407] Connection header received
0810 03:44:51.575790 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.575793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.575831 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.575875 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.576377 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.577061 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.578230 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.578613 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.578699 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.590177 (+ 11478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:51.590185 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:51.590189 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:51.590221 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:51.592087 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.592718 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.592724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.592728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.592829 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.593174 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.593179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.593182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.593513 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:51.593640 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:51.594812 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0810 03:44:51.594949 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 03:44:51.596279 14862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:44:51.599363 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.588449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50036 (local address 127.21.175.125:38491)
0810 03:44:51.588988 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.588999 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.589014 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:44:51.589053 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.589056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.589097 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.589183 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.590055 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.590861 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.591983 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.592326 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.592454 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.595289 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:51.595293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:51.595295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:51.595320 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:51.597106 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.597632 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.597636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.597637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.597734 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.598028 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.598031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.598033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.598250 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:51.598419 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:51.599033 (+ 614us) server_negotiation.cc:299] Negotiation successful
0810 03:44:51.599205 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":125,"threads_started":1}
I20260810 03:44:51.599526 14702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } }
I20260810 03:44:51.600134 14702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fa1a3132d7a42a6ac4de30c10769db4, State: Initialized, Role: FOLLOWER
I20260810 03:44:51.600497 14702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } }
I20260810 03:44:51.602103 14702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:51.602941 22205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Aug 10 03:44:51 dist-test-slave-ss4n krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.175.124@KRBTEST.COM for kudu/127.21.175.124@KRBTEST.COM
I20260810 03:44:51.604225 14673 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4fa1a3132d7a42a6ac4de30c10769db4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37112
I20260810 03:44:51.604463 14673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 03:44:51.604609 14673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:51.605834 14673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } }
I20260810 03:44:51.606263 14781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6: No bootstrap required, opened a new log
I20260810 03:44:51.606303 14874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } } }
I20260810 03:44:51.606544 14874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:44:51.606899 14673 leader_election.cc:288] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd8f428b1e94d91a36dccd9aea11df6 (127.21.175.125:38491), 9ca626539aba4d098275bde40cbc9f6e (127.21.175.124:42341)
I20260810 03:44:51.612203 14781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } }
I20260810 03:44:51.612483 14781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd8f428b1e94d91a36dccd9aea11df6, State: Initialized, Role: FOLLOWER
I20260810 03:44:51.612879 14781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } }
I20260810 03:44:51.613653 14892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } } }
I20260810 03:44:51.614094 14892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:44:51.614578 14781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:51.614650 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4cd8f428b1e94d91a36dccd9aea11df6"
I20260810 03:44:51.614863 14751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:51.612527 14626 master_runner.cc:186] Error getting master registration for 127.21.175.125:38491: OK, instance_id { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" instance_seqno: 1786333491313783 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:44:51.616771 14626 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:44:51.616797 14751 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fa1a3132d7a42a6ac4de30c10769db4 in term 1.
I20260810 03:44:51.617422 14645 leader_election.cc:302] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd8f428b1e94d91a36dccd9aea11df6, 4fa1a3132d7a42a6ac4de30c10769db4; no voters:
I20260810 03:44:51.617815 14874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:44:51.618031 14874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [term 1 LEADER]: Becoming Leader. State: Replica: 4fa1a3132d7a42a6ac4de30c10769db4, State: Running, Role: LEADER
I20260810 03:44:51.618366 14874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } }
I20260810 03:44:51.620129 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.596776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.175.124:42341 (local address 127.0.0.1:49534)
0810 03:44:51.596899 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 03:44:51.596917 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:44:51.596991 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:44:51.597305 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:44:51.597308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:44:51.597524 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:44:51.597658 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.597668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.598791 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.598797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.600019 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:44:51.600034 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.601094 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.601100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:44:51.601207 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.602167 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:44:51.602187 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:44:51.604029 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:44:51.618260 (+ 14231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.618266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.618270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.618600 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.618906 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:44:51.618911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:44:51.618916 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:44:51.618997 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:44:51.619487 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:44:51.619492 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:44:51.619599 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 03:44:51.619709 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 03:44:51.619884 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 03:44:51.620151 14874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fa1a3132d7a42a6ac4de30c10769db4. Latest consensus state: current_term: 1 leader_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } } }
I20260810 03:44:51.620272 14874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [sys.catalog]: This master's current role is: LEADER
W20260810 03:44:51.620446 14890 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:44:51.620503 14890 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:44:51.620694 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.597024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.21.175.124:42341)
0810 03:44:51.597139 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.597142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.597151 (+ 9us) server_negotiation.cc:407] Connection header received
0810 03:44:51.597176 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.597177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.597208 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.597293 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.597789 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.598663 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.600319 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.600856 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.600974 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.604184 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:51.604190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:51.604196 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:51.604226 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:51.617883 (+ 13657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.618723 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.618728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.618732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.618811 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.619088 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.619094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.619098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.619351 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:51.619449 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:51.620350 (+ 901us) server_negotiation.cc:299] Negotiation successful
0810 03:44:51.620479 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:44:51.623978 14862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e: Bootstrap starting.
W20260810 03:44:51.624635 14905 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:44:51.624706 14905 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:44:51.626823 14862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e: Neither blocks nor log segments found. Creating new log.
I20260810 03:44:51.627736 14862 log.cc:824] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:44:51.626520 14709 master_runner.cc:186] Error getting master registration for 127.21.175.124:42341: OK, instance_id { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" instance_seqno: 1786333491540587 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:44:51.630774 14709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:44:51.635222 14909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:44:51.636518 14909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:44:51.643214 14862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e: No bootstrap required, opened a new log
I20260810 03:44:51.644830 14751 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6 [term 1 FOLLOWER]: Refusing update from remote peer 4fa1a3132d7a42a6ac4de30c10769db4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:44:51.645923 14874 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:44:51.646718 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.609332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.21.175.126:44865)
0810 03:44:51.609636 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.609641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.620885 (+ 11244us) server_negotiation.cc:407] Connection header received
0810 03:44:51.628221 (+ 7336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.628225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.628280 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.628335 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.629431 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.630233 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.631554 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.631875 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.631975 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.634995 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:51.634999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:51.635002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:51.635032 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:51.637026 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.643974 (+ 6948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.643979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.643982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.644211 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.645031 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.645035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.645037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.645304 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:51.645457 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:51.646375 (+ 918us) server_negotiation.cc:299] Negotiation successful
0810 03:44:51.646550 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260810 03:44:51.648159 14862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } }
I20260810 03:44:51.648461 14862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca626539aba4d098275bde40cbc9f6e, State: Initialized, Role: FOLLOWER
I20260810 03:44:51.648875 14862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } }
I20260810 03:44:51.649747 14916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } } }
I20260810 03:44:51.650362 14916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:44:51.652952 14892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fa1a3132d7a42a6ac4de30c10769db4. Latest consensus state: current_term: 1 leader_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } } }
I20260810 03:44:51.653116 14892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:44:51.655404 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.632077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.21.175.124:42341)
0810 03:44:51.632222 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.632228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.637773 (+ 5545us) server_negotiation.cc:407] Connection header received
0810 03:44:51.637911 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.637915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.637966 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.638054 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.645316 (+ 7262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.646199 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.647571 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.647912 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.648024 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.650137 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:51.650144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:51.650151 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:51.650189 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:51.652329 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.652965 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.652969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.652971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.653074 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.653470 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.653527 (+ 57us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.653530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.653948 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:51.654205 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:51.654868 (+ 663us) server_negotiation.cc:299] Negotiation successful
0810 03:44:51.655090 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 03:44:51.658564 14862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:44:51.658864 14835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9ca626539aba4d098275bde40cbc9f6e"
I20260810 03:44:51.659054 14834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:44:51.659373 14906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } } }
I20260810 03:44:51.659561 14906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:51.660667 14915 mvcc.cc:205] Tried to move back new op lower bound from 7316821981763969024 to 7316821981672271872. Current Snapshot: MvccSnapshot[applied={T|T < 7316821981763969024}]
I20260810 03:44:51.660697 14834 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e [term 1 FOLLOWER]: Refusing update from remote peer 4fa1a3132d7a42a6ac4de30c10769db4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:44:51.660965 14917 mvcc.cc:205] Tried to move back new op lower bound from 7316821981763969024 to 7316821981672271872. Current Snapshot: MvccSnapshot[applied={T|T < 7316821981763969024}]
I20260810 03:44:51.662221 14874 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:44:51.664268 14835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fa1a3132d7a42a6ac4de30c10769db4 in term 1.
I20260810 03:44:51.665324 14909 catalog_manager.cc:1347] Generated new cluster ID: 719b8981362c4e3d98448c25cf4badc3
I20260810 03:44:51.665400 14909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:44:51.666051 14892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } } }
I20260810 03:44:51.666290 14892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:44:51.672694 14916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fa1a3132d7a42a6ac4de30c10769db4. Latest consensus state: current_term: 1 leader_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } } }
I20260810 03:44:51.672850 14916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:44:51.672959 14916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } } }
I20260810 03:44:51.673043 14916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:44:51.675932 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:51.611804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50042 (local address 127.21.175.125:38491)
0810 03:44:51.612369 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.612376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.632520 (+ 20144us) server_negotiation.cc:407] Connection header received
0810 03:44:51.632644 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.632647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.632698 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.632769 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.638830 (+ 6061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.639634 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.642894 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:51.643253 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:51.643362 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:51.660899 (+ 17537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:44:51.660903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:44:51.660907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:44:51.660945 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:44:51.665794 (+ 4849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.674115 (+ 8321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.674120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.674122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.674236 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:44:51.674708 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:44:51.674712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:44:51.674714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:44:51.674991 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:44:51.675134 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:51.675563 (+ 429us) server_negotiation.cc:299] Negotiation successful
0810 03:44:51.675730 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474}
W20260810 03:44:51.677316 14932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:44:51.677471 14932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:44:51.707724 14874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fa1a3132d7a42a6ac4de30c10769db4" member_type: VOTER last_known_addr { host: "127.21.175.126" port: 44865 } } peers { permanent_uuid: "4cd8f428b1e94d91a36dccd9aea11df6" member_type: VOTER last_known_addr { host: "127.21.175.125" port: 38491 } } peers { permanent_uuid: "9ca626539aba4d098275bde40cbc9f6e" member_type: VOTER last_known_addr { host: "127.21.175.124" port: 42341 } } }
I20260810 03:44:51.707921 14874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4 [sys.catalog]: This master's current role is: LEADER
I20260810 03:44:51.713581 14909 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:44:51.715116 14909 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:44:51.733613 14909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fa1a3132d7a42a6ac4de30c10769db4: Generated new TSK 0
I20260810 03:44:51.734738 14909 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:44:51.813197 14865 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:51.803064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49546 (local address 127.21.175.124:42341)
0810 03:44:51.803206 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.803210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.810924 (+ 7714us) server_negotiation.cc:407] Connection header received
0810 03:44:51.811866 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.811870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.811955 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.812036 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.813039 (+ 1003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49546: BlockingRecv error: recv got EOF from 127.0.0.1:49546 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 03:44:51.813761 14872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:51.802042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50058 (local address 127.21.175.125:38491)
0810 03:44:51.802214 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.802219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.810706 (+ 8487us) server_negotiation.cc:407] Connection header received
0810 03:44:51.811153 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.811157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.811212 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.811267 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.813633 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50058: BlockingRecv error: recv got EOF from 127.0.0.1:50058 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 03:44:51.815474 14706 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:51.809952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37132 (local address 127.21.175.126:44865)
0810 03:44:51.810084 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.810089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.814407 (+ 4318us) server_negotiation.cc:407] Connection header received
0810 03:44:51.814633 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.814637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.814691 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.814742 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.815353 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37132: BlockingRecv error: recv got EOF from 127.0.0.1:37132 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 03:44:51.839257 14872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:51.822691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.21.175.125:38491)
0810 03:44:51.822804 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.822809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.838009 (+ 15200us) server_negotiation.cc:407] Connection header received
0810 03:44:51.838318 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.838323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.838406 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.838466 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.839135 (+ 669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50066: BlockingRecv error: recv got EOF from 127.0.0.1:50066 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 03:44:51.855116 14872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:51.846262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50078 (local address 127.21.175.125:38491)
0810 03:44:51.846382 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.846387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.853971 (+ 7584us) server_negotiation.cc:407] Connection header received
0810 03:44:51.854222 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.854226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.854291 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.854360 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.854994 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50078: BlockingRecv error: recv got EOF from 127.0.0.1:50078 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 03:44:51.867406 14872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:51.860753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.21.175.125:38491)
0810 03:44:51.860877 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.860881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.866275 (+ 5394us) server_negotiation.cc:407] Connection header received
0810 03:44:51.866498 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.866516 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.866577 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.866644 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.867272 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50084: BlockingRecv error: recv got EOF from 127.0.0.1:50084 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 03:44:51.887235 14872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:51.876946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.21.175.125:38491)
0810 03:44:51.877130 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.877134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.885994 (+ 8860us) server_negotiation.cc:407] Connection header received
0810 03:44:51.886278 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.886283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.886353 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.886418 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.887101 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50094: BlockingRecv error: recv got EOF from 127.0.0.1:50094 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260810 03:44:51.899993 14872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:51.897991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37572 (local address 127.21.175.125:38491)
0810 03:44:51.898126 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.898130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.898851 (+ 721us) server_negotiation.cc:407] Connection header received
0810 03:44:51.899098 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.899101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.899168 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.899227 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.899867 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37572: BlockingRecv error: recv got EOF from 127.0.0.1:37572 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 03:44:51.927383 14872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:51.925965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37584 (local address 127.21.175.125:38491)
0810 03:44:51.926106 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.926111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.926411 (+ 300us) server_negotiation.cc:407] Connection header received
0810 03:44:51.926604 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.926610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.926677 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.926771 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.927261 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37584: BlockingRecv error: recv got EOF from 127.0.0.1:37584 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 03:44:51.963258 14872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:51.961787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37588 (local address 127.21.175.125:38491)
0810 03:44:51.962005 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:51.962014 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:51.962064 (+ 50us) server_negotiation.cc:407] Connection header received
0810 03:44:51.962312 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:51.962317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:51.962388 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:51.962505 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:51.963117 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37588: BlockingRecv error: recv got EOF from 127.0.0.1:37588 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 03:44:52.030990 14872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:52.029633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37602 (local address 127.21.175.125:38491)
0810 03:44:52.029833 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:52.029838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:52.030025 (+ 187us) server_negotiation.cc:407] Connection header received
0810 03:44:52.030213 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:52.030217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:52.030276 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:52.030369 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:52.030867 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37602: BlockingRecv error: recv got EOF from 127.0.0.1:37602 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260810 03:44:52.162927 14872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:52.161341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37618 (local address 127.21.175.125:38491)
0810 03:44:52.161477 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:52.161516 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:52.161942 (+ 426us) server_negotiation.cc:407] Connection header received
0810 03:44:52.162087 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:52.162090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:52.162155 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:52.162238 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:52.162801 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37618: BlockingRecv error: recv got EOF from 127.0.0.1:37618 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 03:44:52.422678 14872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:44:52.421357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37626 (local address 127.21.175.125:38491)
0810 03:44:52.421543 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:52.421548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:52.421688 (+ 140us) server_negotiation.cc:407] Connection header received
0810 03:44:52.421887 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:52.421890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:52.421939 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:52.422059 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:44:52.422548 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37626: BlockingRecv error: recv got EOF from 127.0.0.1:37626 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:44:52.626012 14905 catalog_manager.cc:1259] Loaded cluster ID: 719b8981362c4e3d98448c25cf4badc3
I20260810 03:44:52.626122 14905 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6: loading cluster ID for follower catalog manager: success
I20260810 03:44:52.627869 14905 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6: acquiring CA information for follower catalog manager: success
I20260810 03:44:52.629091 14905 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4cd8f428b1e94d91a36dccd9aea11df6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:44:52.678740 14932 catalog_manager.cc:1259] Loaded cluster ID: 719b8981362c4e3d98448c25cf4badc3
I20260810 03:44:52.678870 14932 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e: loading cluster ID for follower catalog manager: success
I20260810 03:44:52.680753 14932 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e: acquiring CA information for follower catalog manager: success
I20260810 03:44:52.682122 14932 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9ca626539aba4d098275bde40cbc9f6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:44:52.941295 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:52.936900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.21.175.125:38491)
0810 03:44:52.937151 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:52.937155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:52.937260 (+ 105us) server_negotiation.cc:407] Connection header received
0810 03:44:52.937382 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:52.937384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:52.937543 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:52.937696 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:44:52.938023 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:52.938539 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:52.939593 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:52.939810 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:52.939892 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:52.940319 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:44:52.940805 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:44:52.940908 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:52.941106 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 03:44:52.941167 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":97,"threads_started":1}
I20260810 03:44:52.948704 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:52.944782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43826 (local address 127.21.175.124:42341)
0810 03:44:52.945006 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:52.945010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:52.945029 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:44:52.945144 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:52.945146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:52.945266 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:52.945366 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:44:52.945637 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:52.946172 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:52.947150 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:52.947381 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:52.947514 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:52.947831 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:44:52.948274 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:44:52.948375 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:52.948463 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 03:44:52.948568 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
I20260810 03:44:52.956812 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:52.952193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43834 (local address 127.21.175.124:42341)
0810 03:44:52.952316 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:52.952321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:52.952719 (+ 398us) server_negotiation.cc:407] Connection header received
0810 03:44:52.952831 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:52.952834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:52.952890 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:52.952960 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:44:52.953289 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:52.953873 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:52.955329 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:52.955704 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:52.955832 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:52.956187 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:44:52.956290 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:44:52.956397 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:52.956581 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 03:44:52.956644 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 03:44:52.957252 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:52.952004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37642 (local address 127.21.175.125:38491)
0810 03:44:52.952139 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:52.952143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:52.952430 (+ 287us) server_negotiation.cc:407] Connection header received
0810 03:44:52.952719 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:52.952724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:52.952798 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:52.952884 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:44:52.953179 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:52.954058 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:52.955374 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:52.955723 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:52.955832 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:52.956275 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:44:52.956380 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:44:52.956464 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:52.957071 (+ 607us) server_negotiation.cc:299] Negotiation successful
0810 03:44:52.957138 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 03:44:52.957454 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:44:52.952143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54824 (local address 127.21.175.126:44865)
0810 03:44:52.952439 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0810 03:44:52.952444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:44:52.952860 (+ 416us) server_negotiation.cc:407] Connection header received
0810 03:44:52.953128 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:44:52.953135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:44:52.953317 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:44:52.953447 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:44:52.953859 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:52.954550 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:52.955565 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:44:52.955945 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:44:52.956055 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:44:52.956471 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:44:52.956883 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:44:52.957023 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 03:44:52.957202 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 03:44:52.957272 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260810 03:44:52.961058 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 14626
I20260810 03:44:52.973222 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 14709
I20260810 03:44:52.982802 22205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMmCxq6/build/debug/bin/kudu with pid 14789
2026-08-10T03:44:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3381 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6626 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (302945 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST